In case of CAPTCHA raise a SearxEngineCaptchaException and suspend for 7 days.
When get_sc_code() fails raise a SearxEngineResponseException and suspend for 7
days.
[1] https://github.com/searxng/searxng/pull/695
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Startpage has introduced new anti-scraping measures that make SearXNG instances
run into captchas:
1. some arguments has been removed and a new `sc` has been added.
2. search path changed from `do/search` to `sp/search`
3. POST request is no longer needed
Closes: https://github.com/searxng/searxng/issues/692
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
api.openverse.engineering is a little picky and wants to have a trailing slash
in the path:
/v1/images? -->/ v1/images/?
otherwise it redirects, here is the debug log:
DEBUG searx.network.openverse : HTTP Request: GET https://api.openverse.engineering/v1/images?&page=1&page_size=20&format=json&q=foo "HTTP/2 301 Moved Permanently" (text/html; charset=utf-8)
DEBUG searx.network.openverse : HTTP Request: GET https://api.openverse.engineering/v1/images/?&page=1&page_size=20&format=json&q=foo "HTTP/2 200 OK" (application/json)
WARNING searx.engines.openverse : ErrorContext('searx/search/processors/online.py', 105, 'count_error(', None, '1 redirects, maximum: 0', ('200', 'OK', 'api.openverse.engineering')) True
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The implementation of the etools engine is poor. No date-range support, no
language support and it is broken by a CAPTCHA.
etools is a metasearch engine, the major search engines it supports (google,
bing, wikipedia, Yahoo) are already available in SeaarXNG.
While etools does support several engines we currently don't support directly,
support for them should be added directly to SearXNG if there is demand.
In practice: in SearXNG the worse etools results will be mixed with good results
from other engines we have (as long as there is no captcha).
At best case, what we win with etools is in e.g. results from de.ask.com in a
query from a german request .. in all other cases worse results are bubble up in
SearXNG's result list.
[1] https://github.com/searxng/searxng/issues/696#issuecomment-1005855499
Closes: https://github.com/searxng/searxng/issues/696
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The previous implementation used two hash sets and a list.
... that's not necessary ... a single hash map suffices.
And it's also less error prone ... because the previous data structure
allowed a setting to be enabled and disabled at the same time.
Previously the default_value was abused for the cookie name.
Having SwitchableSetting subclass Setting doesn't even make sense
in the first place since none of the Setting methods apply.
The ? search operator has been broken for some time and
currently only raises the question why it's still there.
## Context ##
The query "Paris !images" searches for "Paris" in the "images" category.
Once upon a time Searx supported "Paris ?images" to search for "Paris"
in the currently enabled categories and the "images" category.
The feature makes sense ... the ? syntax does not.
We will hopefully introduce a +!images syntax in the future.
Fixes#702.
* allow not to record metrics (response time, etc...)
* this commit doesn't change the UI. If the metrics are disabled
/stats and /stats/errors will return empty response.
in /preferences, the columns response time and reliability will be empty.
The tab icon names are currently hard coded in the templates.
This commit lets us introduce an icon property in the future, e.g:
categories_as_tabs:
general:
icon: search-outline
These dictionaries are no longer part of the general category,
so they're no longer queried by default -> we can enable them
by default without degrading general query performance.
The general category is the category that is searched by default.
From a privacy standpoint it doesn't make sense to send all general
queries to specialized search engines that cannot deal with those
queries anyway.
Previously we didn't have a good place to put search engines that don't
fit into any of the tab categories. This commit automatically puts
search engines that don't belong to any tab category in an "other"
category, that is only displayed in the user preferences (and not above
search results).
Add a redis connector, the default DB connector is a socket at::
unix:///usr/local/searxng-redis/run/redis.sock?db=0
To set up a redis instance simply use::
$ ./manage redis.build
$ sudo -H ./manage redis.install
A hint for developers:
To get access rights to this instance, your developer account needs to be added
to the *searxng-redis* group::
$ sudo -H ./manage redis.addgrp "${USER}"
# don't forget to logout & login to get member of group
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Previously all categories were displayed as search engine tabs.
This commit changes that so that only the categories listed under
categories_as_tabs in settings.yml are displayed.
This lets us introduce more categories without cluttering up the UI.
Categories not displayed as tabs can still be searched with !bangs.
Add event listener to query selector::
'#urls img.image'
From the user point of view, I think it is better to hide the image:
img_load_error.svg is helplful in the image category because it still allows to
select the image. IMO, in the news category, the fact there is a missing image
won't help to choose the links. From a developer point of view, the place holder
is signal that may be the engine needs to be updated (at least give a look). The
browser console should show the same information too, but it requires some
additional steps. [1]
[1] https://github.com/searxng/searxng/pull/610#issuecomment-997640132
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Fix pylint issues from commit (3d96a983)
[format.python] initial formatting of the python code
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Disable the python code formatting from python-black, where the readability of
code suffers by formatting.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Follow up queries for the pages needed to be fixed.
- Split search-term in one for initial query and one for following queries.
- Set some headers in HTTP requests, bing needs for paging support.
- IMO //div[@class="sa_cc"] does no longer match in a bing response.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Fix remarks from pylint and improved code-style. In preparation for a bug-fix
of the Bing (Web) engine I add this engine to the pylint-list.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Images should include dimension attributes. Without `width` and `height`
specified, image dimensions are 0×0 pixels at first. ... In this case the
browser determines that all of them are visible to the user and decides to load
everything [1].
In CSS the `width` is set to a value and the `height` is unsed to scale the image
proportional in both dimensions.
[1] https://web.dev/browser-level-image-lazy-loading/#images-should-include-dimension-attributes
[2] https://caniuse.com/loading-lazy-attr
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::
make themes.simple
make themes.simple.test
In the github CI the `themes.all` target enforce a `themes.simple`.
BTW: Remove 'jshint' left overs from 0ee316f3d
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
automatically fix some of the problems reported by eslint rules::
$ ./manage nvm.bash
nvm-env$ npm --prefix searx/static/themes/simple run eslint-fix
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
All three filters (`language`, `time_range` and `safesearch`) are rendered in
one line. A size of 45% for `language` and `time_range` left only 10% for the
`safesearch` filter. Solution: drop with from `language` and `time_range`.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The CSP issue is, that the `_Position` function in the autocomplete-js set the
style attributes by `setAttribute("style", ...)`. Using `setAttribute` to set
the style attribute invokes the HTML parser and CSP is triggered [1].
This patch overwrite the `_Position` function of autocomplete-js.
BTW: remove trailing whitespace
[1] https://stackoverflow.com/a/57633533
Closes: https://github.com/searxng/searxng/issues/352
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Include the source files directly in the `.map` files [1] for two reasons:
1. In the simple theme, modules from `./node_modules` are only available on
developer's build host (a `make node.env` is needed to get the sources). By
example; in the `searxng.min.js.map` file, the `autocomplete.js` is linked at
URL:
../node_modules/autocomplete-js/dist/autocomplete.js
2. Later, when client sources has been moved out of the `./static` URL [2] to
the `<repo-roo>/src` folder, all JS sources are no longer available.
Drawback: `.map` files grow up in size, but this should not bother, because the
files are only loaded in debug mode.
[1] https://github.com/gruntjs/grunt-contrib-uglify#sourcemapincludesources
[2] https://github.com/searxng/searxng/issues/141
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
remove:
* searx/static/themes/simple/js/searxng.head.js
* searx/static/themes/simple/js/searxng.js
* searx/static/themes/simple/css/searxng-rtl.css
* searx/static/themes/simple/css/searxng.css
These files are never used.
About the js files: the .map files references the sources instead of the concated version.
In window.searxng.theme the theme data can be passed through to JS
implementations.
Initial the window.searxng.theme namespace starts with a value for
`img_load_error`::
// image that is displayed if load of <img src='...'> failed
img_load_error: 'img/img_load_error.svg'
The searx/static/themes/__common__/js/image_layout.js is patched to uses the
value, if the theme defines a value for img_load_error in this namespace.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The *background color* of an transparent image is the color of the color of
wallpaper on which all images are shown on.
Set variable `--color-result-image-background` and `--color-base-background` to
the same value.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* move `searx/static/themes/simple/img/searxng.svg` to `src/brand/searxng.svg`
* README.rst can use it without a reference to a theme.
* the simple theme can create `searx/static/themes/simple/img/searxng.png` using
the svg2png task
Suggested-by: @dalf https://github.com/searxng/searxng/pull/561#issuecomment-981747902
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This is a follow up of 9a3253fc escaping `<` and `>` in all themes. This patch
fix issue in oscar theme
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
define a custom grunt task, since grunt-sharp is too old (it can't be installed).
in gruntfile.js, the image tasks are moved at the end the build chain.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- remove unused 'path' constant
- fix indentation
- Put all image tasks at the end, where it is easy to list which image is processed.
Suggested -by: @dalf https://github.com/searxng/searxng/pull/541#discussion_r756247157
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the video search, google also sometimes includes news. E.g. in the DE
language when you search for `!gov paris`, google adds an article from a german
newspaper (FAZ), I assume these are sponsored link (not tagged advertisement?)
Those links do not have an image / this patch ignores *video links* wqithout an
image ID.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Strings like::
'Query in the page\'s title'
are hard to read / remove escape sequence by using double quotation marks for
strings ::
"Query in the page's title"
BTW: remove a leading dot in the simple theme [1].
[1] 80fb77476f (r756112716)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Support for the prefixed version (-moz-border-radius) was removed in Gecko
13.0 (Firefox 13.0 / Thunderbird 13.0 / SeaMonkey 2.10) [1].
[1] https://developer.mozilla.org/pt-BR/docs/Web/CSS/border-radius
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* rework selection UI in pref (fix based on: 78643e9f43)
* moved search filters underneath categories
* cut params from url and replace / with ›
* make h3 and url in article bigger
* add safe search select to search filter (this will not override settings and only be valid while on result page in a session)
* make search form button not overlap each other when js is disabled
* 1rem padding around preview image and thumbnail in default article template
Sometimes there is no href in the `<a ..>` tag of a *link_node* [1].
[1] https://github.com/searxng/searxng/issues/532
Reported-by: @TheEssem
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Folder src/brand is intended to place logo, wordmark and more in. This commit
moves the origin (source) of searxng-wordmark.svg from the simple theme into
this folder
About the new created src/ folder:
On the long term we can move all the files from static/themes/<name>/src/* into
a src/themes/<name>/* folder.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since digg no longer works, we do nat have a active engine in the social-media
category. Enable reddit by default to have at least one engine back in this
category.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
when on the top of the page, the image is just below the search fields.
this commit slightly aligh the top of the image detail with the bottom
of the search fields.
the width of #main_results #results.image-detail-open.only_template_images
was set to 59.25rem. On mobile, the images overflowed on right side of
the viewport, which creates a horizontal scroll.
This commit set the value to min(98%, 59.25rem) whatever the max-width is.
* update from ionicons-npm to ionicons ver.5
* drop the webfont built by grunt for icons
* built icons.html template for inlining icons with jinja2 into html
* update icon to use mostly the outline version
* add icons to categories and do not display them on mobile to save space
* remove all legacy ion icon font files from simple theme
* icons.html is added in this commit since make statc.build.restore requires git to know the file already
* cleanup error-dialog
new bash function convert_if_newer() / usage::
convert_if_newer <origfile> <outfile> [<options>, ...]
convert_if_newer "path/to/origin.svg" "path/to/converted.png" -transparent white -resize 64x64
Run's ImageMagik' convert comand to generate <outfile> from <origfile>, if
<origfile> is newer than <outfile>. The command line is to convert is::
convert <origfile> [<options>, ...] <outfile>
PNG 'searx/static/themes/simple/img/favicon.png' has been created by::
$ make themes.simple
CONVERT searx/static/themes/simple/src/svg/searxng-wordmark.svg -transparent white -resize 64x64 searx/static/themes/simple/img/favicon.png
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* remove hover effect from vim help modal
* remove bold font weight from active item in prefenreces tab bar
* remove margin from answer box to unify with rest of result page
* disable by default
* settings.yml: ui.query_in_title
* in /preferences: privacy tab
when enabled, the result page's title contains the user query.
previously:
* oscar theme: the query was always included
* simple theme: the query was included with the GET method
Before this commit, the default click event on an image result is prevented,
this include clicks inside the detail.
This commit makes sure the click happends outside the detail to prevent the default event.
buttons:
* previous & next in the result page
* save, restore, & back in the preferences
* back to top
<select> input in Chrom* browsers:
* fix the white text with a white background issue
It is not possible to use CSS variable in a SVG when this is in a background.
This commit adds two .svg files, less converts them into data URL.
The two files are indentical except the fill color.
* remove vim arrow
* add 1rem padding to results
* add 0.2rem left border to vim selected article
* set column gap to 1.2rem and make search bar in line with results
* put 10px border-radius selected article
* result article: 0.125rem margin on tablet and esktop; 1rem margin on phone
* index page: margin top is 24% of the view port.
avoid to scroll a small screen,
center the content in the middle of the screen
* link to preferences at the same height same the input fields
* increase the category tab heighs
* increase the margin bottom of the query field
* in the results, change the h3 margin top and bottom to 0.4rem (6px)
* move the back to top button slightly on the right when the results are only images
When an image is selected, the detail with the full size image is displayed
on the right side of the screen (or full screen on tablet and phone).
When Javascript is disabled, the thumbnail is a linked to the full size image,
as it was before.
When the image proxy is enabled, the full size image is also proxied,
in consequence this commit increases the bandwidth usage of instances.
The detail can be closed by the close button or the Esc key.
It is possible to go to the next and previous images using the j and k keys
or the button on the top right of the screen.
* url in article head is clickable
* url is bigger now 0.96em font
* url is now left floating on tablet and phone
* there is a 8px top and bottom margin on h3 result heading
* rework search form as grid layout
* remove various paddings and margins
* add logo and link to index to search form
* make categories bigger on phone
Markus Heiser
Martin Fischer
Martin Minka
Léon Tiekötter
searxng-bot
Alexandre Flament
dalf
mrpaulblack
dependabot[bot]
Tom
Igor Rzegocki
MrPaulBlack