Commit Graph

4754 Commits

Author SHA1 Message Date
Markus Heiser 1ec325adcc [mod] limiter -> botdetection: modularization and documentation
In order to be able to meet the outstanding requirements, the implementation is
modularized and supplemented with documentation.

This patch does not contain functional change, except it fixes issue #2455

----

Aktivate limiter in the settings.yml and simulate a bot request by::

    curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \
         -H 'Accept: text/html'
         -H 'User-Agent: xyz' \
         -H 'Accept-Encoding: gzip' \
         'http://127.0.0.1:8888/search?q=foo'

In the LOG:

    DEBUG   searx.botdetection.link_token : missing ping for this request: .....

Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time
before you get a "Too Many Requests" response.

Closes: https://github.com/searxng/searxng/issues/2455
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
Markus Heiser 5226044c13 [mod] limiter: add random token to the limiter URL
By adding a random component in the limiter URL a bot can no longer send a ping
by request a static URL.

Related: https://github.com/searxng/searxng/pull/2357#issuecomment-1518525094
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
Markus Heiser dba569462d [mod] limiter: reduce request rates for requests without a ping
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
dalf c1b5ff7e1c Update searx.data - update_engine_descriptions.py 2023-05-29 07:28:50 +02:00
dalf 2ba50d392e Update searx.data - update_currencies.py 2023-05-29 07:28:18 +02:00
dalf cb843ef13c Update searx.data - update_engine_traits.py 2023-05-29 07:27:40 +02:00
dalf 512e001277 Update searx.data - update_firefox_version.py 2023-05-29 07:27:07 +02:00
dalf f03ac9b152 Update searx.data - update_wikidata_units.py 2023-05-29 07:26:47 +02:00
dalf e12e350f7f Update searx.data - update_ahmia_blacklist.py 2023-05-29 07:26:20 +02:00
Markus Heiser 3ca97cf5e3 [fix] simple theme: move engine alerts in case of no results into sidebar
If there were no results but errors in the engines then the error dialogs of the
engines was displayed in the result list.

With the new design errors of the engines should only be displayed in the
sidebar and at the same time duplications of the (template) code will be
avoided.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-28 12:19:32 +02:00
mrpaulblack 60b94dfdca [build] /static 2023-05-28 12:19:32 +02:00
mrpaulblack f087959b02 [mod] simple theme: build design for details (collapsables)
* set border top and bottom on sidebar collasables
* inrease peading on summary so its easier to click on mobile
* remove margins and add flex wrapper to normalize elements in sidebar
2023-05-28 12:19:32 +02:00
Markus Heiser b7e315563d [mod] simple theme: collaps/expand elements in the sidebar
Make elements in the sidebar collapse able.  Except infoboxes all elements in
the sidebar are collapsed by default.

By folding out the sidebar elements, the UI looks less cluttered.  Especially on
small devices like smartphones, where the sidebar is above the results list, the
UX should be improved [1].

[1] https://github.com/searxng/searxng/issues/2140

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-28 12:19:32 +02:00
searxng-bot e8054026fb [translations] update from Weblate
69171f12 - 2023-05-25 - fabiosantoscode <fabiosantosart@gmail.com>
2caaed0a - 2023-05-23 - trmx <borcan.cristian1@gmail.com>
84d1702b - 2023-05-21 - return42 <markus.heiser@darmarit.de>
65cc6eb8 - 2023-05-21 - return42 <markus.heiser@darmarit.de>
e0ab3383 - 2023-05-22 - return42 <markus.heiser@darmarit.de>
23e87f15 - 2023-05-21 - return42 <markus.heiser@darmarit.de>
14f0fc6b - 2023-05-21 - return42 <markus.heiser@darmarit.de>
5b7c7b7d - 2023-05-21 - return42 <markus.heiser@darmarit.de>
c725b38d - 2023-05-21 - return42 <markus.heiser@darmarit.de>
2023-05-26 07:08:21 +00:00
Markus Heiser ebe22a4319 [fix] typo: dues --> does
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-22 13:18:22 +02:00
Markus Heiser bc647fabaf [fix] ClientPref - don't raise exception if Accept-Language is invalid
If the Accept-Language header [1] is set but empty or holds a value that is
unknown to babel, an excpetion is raised::

    $ curl --header 'Accept-Language: xyz' 'http://127.0.0.1:8888/search?q=foo'
    ...
    Traceback (most recent call last):
      File "searx/preferences.py", line 335, in from_http_request
        return cls(locale=pairs[0][0])
    IndexError: list index out of range

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language

Reported by: @Eolien55 in https://github.com/searxng/searxng/issues/2434#issuecomment-1556199789
Closes: https://github.com/searxng/searxng/issues/2434
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-22 12:38:59 +02:00
Markus Heiser bab730c8a8
Merge pull request #2446 from searxng/translations_update
Update translations
2023-05-19 19:59:58 +02:00
pankaj 4900c091a6 use logger.warning
logger.warn() is depricated.
logger.warning is already being used in some files.
2023-05-19 19:35:29 +05:30
searxng-bot 23b53a03c2 [translations] update from Weblate
2eeec66c - 2023-05-13 - return42 <markus.heiser@darmarit.de>
87058e51 - 2023-05-13 - return42 <markus.heiser@darmarit.de>
2023-05-19 07:07:56 +00:00
Markus Heiser 007a615ffa [mod] donation_url: disable by default
SearXNG's donation campaign has been ended.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-15 09:19:17 +02:00
Markus Heiser 6625b6193e
Merge pull request #2420 from searxng/translations_update
Update translations
2023-05-13 06:06:07 +02:00
Markus Heiser caebd297e9 [fix] engine ddg: minor change in the API of ddg
Closes: https://github.com/searxng/searxng/issues/2419
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-12 18:58:49 +02:00
searxng-bot f5c407eaa0 [translations] update from Weblate
df7e1be3 - 2023-05-10 - return42 <markus.heiser@darmarit.de>
7ae9877e - 2023-05-08 - return42 <markus.heiser@darmarit.de>
c2fe5131 - 2023-05-07 - KDesp73 <kdesp2003@gmail.com>
2023-05-12 07:07:40 +00:00
Markus Heiser a60851bd59 [fix] version format string generated by 'git show'
Newer versions of git [1] do no longer support a format string that includes a minus
to remove leading zeros [2].  The format string '%Y.%m.%d'  is more version rod.

[1] https://github.com/searxng/searxng/issues/2413#issuecomment-1542320387
[2] https://github.com/searxng/searxng/pull/2122/files

Closes: https://github.com/searxng/searxng/issues/2413
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-10 18:33:45 +02:00
Markus Heiser db95c4713b
Merge pull request #2406 from searxng/dependabot/npm_and_yarn/searx/static/themes/simple/master/grunt-contrib-cssmin-5.0.0
Bump grunt-contrib-cssmin from 4.0.0 to 5.0.0 in /searx/static/themes/simple
2023-05-06 05:54:15 +02:00
dependabot[bot] 673837f83e
Bump grunt-contrib-cssmin in /searx/static/themes/simple
Bumps [grunt-contrib-cssmin](https://github.com/gruntjs/grunt-contrib-cssmin) from 4.0.0 to 5.0.0.
- [Release notes](https://github.com/gruntjs/grunt-contrib-cssmin/releases)
- [Changelog](https://github.com/gruntjs/grunt-contrib-cssmin/blob/main/CHANGELOG)
- [Commits](https://github.com/gruntjs/grunt-contrib-cssmin/compare/v4.0.0...v5.0.0)

---
updated-dependencies:
- dependency-name: grunt-contrib-cssmin
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-05-05 07:57:12 +00:00
searxng-bot 54c4d08167 [translations] update from Weblate
70336613 - 2023-05-03 - return42 <markus.heiser@darmarit.de>
55d82b96 - 2023-05-03 - artnay <jiri.gronroos@iki.fi>
3911fe35 - 2023-05-03 - return42 <markus.heiser@darmarit.de>
81b6ebd1 - 2023-05-03 - return42 <markus.heiser@darmarit.de>
6655ac63 - 2023-05-01 - return42 <markus.heiser@darmarit.de>
3b9cccb8 - 2023-04-30 - return42 <markus.heiser@darmarit.de>
51601c0b - 2023-04-30 - return42 <markus.heiser@darmarit.de>
2023-05-05 07:07:41 +00:00
Markus Heiser 823c490c84 [mod] limiter: block requests from PetalBot
Block requests from PetalBlock.  Normally robots.txt is enough to stop
PetalBlock from making requests [1].  However, if SearXNG is offered below a
path (example.org/search), then the robots.txt is not available in the root
paths of the domain / subdomain.

[1] https://webmaster.petalsearch.com/site/petalbot

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-30 09:49:26 +02:00
Markus Heiser 7b43f98ebb
Merge pull request #2382 from searxng/translations_update
Update translations
2023-04-30 08:11:00 +02:00
Markus Heiser e0c8e1923d
Merge pull request #2390 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2023-04-30 07:16:31 +02:00
Markus Heiser d63dbb10fc
Merge pull request #2391 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2023-04-30 07:15:55 +02:00
Markus Heiser e9fdfab76e
Merge pull request #2392 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2023-04-30 07:15:28 +02:00
Markus Heiser 836827517d
Merge pull request #2393 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2023-04-30 07:14:53 +02:00
Markus Heiser cfc01ea068
Merge pull request #2394 from searxng/update_data_update_engine_traits.py
Update searx.data - update_engine_traits.py
2023-04-30 07:14:23 +02:00
dalf c2fbace534 Update searx.data - update_engine_descriptions.py 2023-04-29 00:24:27 +00:00
dalf 4f31ab7d4b Update searx.data - update_engine_traits.py 2023-04-29 00:14:09 +00:00
dalf df4cc070ec Update searx.data - update_ahmia_blacklist.py 2023-04-29 00:13:21 +00:00
dalf 5b93f97fb2 Update searx.data - update_currencies.py 2023-04-29 00:13:19 +00:00
dalf 7c90a6a222 Update searx.data - update_firefox_version.py 2023-04-29 00:13:09 +00:00
dalf 4336f70b59 Update searx.data - update_wikidata_units.py 2023-04-29 00:13:08 +00:00
searxng-bot 274fcd46b6 [translations] update from Weblate
b6658877 - 2023-04-27 - return42 <markus.heiser@darmarit.de>
d7a3917b - 2023-04-25 - return42 <markus.heiser@darmarit.de>
879248ad - 2023-04-25 - return42 <markus.heiser@darmarit.de>
6ccafe4e - 2023-04-25 - return42 <markus.heiser@darmarit.de>
d202aed8 - 2023-04-23 - Parsa Ranjbar <parsa@disr.it>
2023-04-28 07:08:09 +00:00
Markus Heiser 45529f51a1
Merge pull request #2347 from return42/mod-lang-detection
If language recognition fails use the Accept-Language
2023-04-25 15:46:26 +02:00
Jakub Łukasiewicz 7da47edd93 [mod] external bang: go to main instead of search page when query is empty
Closes: #2368
2023-04-25 15:02:34 +02:00
Markus Heiser 4080eca3dd [build] /static 2023-04-21 13:38:29 +02:00
searxng-bot b41e47eea6 [translations] update from Weblate
72d42638 - 2023-04-15 - tentsbet <remendne@pentrens.jp>
560e1885 - 2023-04-16 - return42 <markus.heiser@darmarit.de>
7370b026 - 2023-04-16 - return42 <markus.heiser@darmarit.de>
20946697 - 2023-04-16 - return42 <markus.heiser@darmarit.de>
2023-04-21 07:07:58 +00:00
Markus Heiser 9b575a997b [fix] doc of locales.get_engine_locale() / zh-classical is missleading
Wikipedia's zh-classical is not zh_Hant (see doc-string of engines.wikipedia).
Fixed the example in the doc-string of locales.get_engine_locale() to 'zh_TW'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-17 08:48:57 +02:00
Markus Heiser f1b6351ae1 [fix] engine: google play movies
Closes: https://github.com/searxng/searxng/pull/1746
Closes: https://github.com/searxng/searxng/issues/1599

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-16 19:15:44 +02:00
Marc Abonce Seguin be1f6ee1c4 Update searx.data - update_engine_traits.py 2023-04-16 08:40:44 +02:00
Markus Heiser 8adbc4fcec [mod] settings.yml: enable language detection by default_lang (auto)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 22:24:59 +02:00
Markus Heiser d5ecda9930 [mod] move language recognition to get_search_query_from_webapp
To set the language from language recognition and hold the value selected by the
client, the previous implementation creates a copy of the SearchQuery object and
manipulates the SearchQuery object by calling function replace_auto_language().

This patch tries to implement a similar functionality in a more central place,
in function get_search_query_from_webapp() when the SearchQuery object is build
up.

Additional this patch uses the language preferred by the client, if language
recognition does not have a match / the existing implementation does not care
about client preferences and uses 'all' in case of no match.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 22:23:33 +02:00
Markus Heiser c03b0ea650 [mod] add a Preferences.client property to store client prefs
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 21:41:57 +02:00
Markus Heiser 6592f0d36e [build] /static 2023-04-15 18:59:46 +02:00
Markus Heiser abf574dc0c [mod] Ignore autocomplete_min on queries that include '!' (!bang)
Closes: https://github.com/searxng/searxng/issues/1566
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 18:54:47 +02:00
Markus Heiser 09295a3fd1 Update searx.data - update_engine_descriptions.py
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 16:04:05 +02:00
Markus Heiser a8fb6dffb2 [upd] make data.traits --> searx/data/engine_traits.json
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 16:04:05 +02:00
Markus Heiser 27369ebec2 [fix] searxng_extra/update/update_engine_descriptions.py (part 1)
Follow up of #2269

The script to update the descriptions of the engines does no longer work since
PR #2269 has been merged.

searx/engines/wikipedia.py
==========================

1. There was a misusage of zh-classical.wikipedia.org:

   - `zh-classical` is dedicate to classical Chinese [1] which is not
     traditional Chinese [2].

   - zh.wikipedia.org has LanguageConverter enabled [3] and is going to
     dynamically show simplified or traditional Chinese according to the
     HTTP Accept-Language header.

2. The update_engine_descriptions.py needs a list of all wikipedias.  The
   implementation from #2269 included only a reduced list:

   - https://meta.wikimedia.org/wiki/Wikipedia_article_depth
   - https://meta.wikimedia.org/wiki/List_of_Wikipedias

searxng_extra/update/update_engine_descriptions.py
==================================================

Before PR #2269 there was a match_language() function that did an approximation
using various methods.  With PR #2269 there are only the types in the data model
of the languages, which can be recognized by babel.  The approximation methods,
which are needed (only here) in the determination of the descriptions, must be
replaced by other methods.

[1] https://en.wikipedia.org/wiki/Classical_Chinese
[2] https://en.wikipedia.org/wiki/Traditional_Chinese_characters
[3] https://www.mediawiki.org/wiki/Writing_systems#LanguageConverter

Closes: https://github.com/searxng/searxng/issues/2330
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 16:03:59 +02:00
Markus Heiser 0adfed195e
Merge pull request #2338 from return42/block-old-farside
[mod] limiter: block unmaintained Farside instances
2023-04-15 08:40:30 +02:00
Markus Heiser 5f6ee2f82c
Merge pull request #2342 from searxng/dependabot/pip/master/pygments-2.15.0
Bump pygments from 2.14.0 to 2.15.0
2023-04-15 08:28:05 +02:00
Markus Heiser b3d8d305d2 [build] /static 2023-04-15 08:21:02 +02:00
searxng-bot a938aeaf1b [translations] update from Weblate
8d1975e8 - 2023-04-12 - return42 <markus.heiser@darmarit.de>
76ff8083 - 2023-04-13 - return42 <markus.heiser@darmarit.de>
a53721ef - 2023-04-13 - return42 <markus.heiser@darmarit.de>
9d8b4810 - 2023-04-08 - Eryk Michalak <gnu.ewm@protonmail.com>
1587a991 - 2023-04-10 - tentsbet <remendne@pentrens.jp>
16c84cef - 2023-04-10 - return42 <markus.heiser@darmarit.de>
29845d20 - 2023-04-09 - ghose <correo@xmgz.eu>
ccdee956 - 2023-04-08 - gallegonovato <fran-carro@hotmail.es>
402b3d27 - 2023-04-08 - return42 <markus.heiser@darmarit.de>
44b1ea99 - 2023-04-10 - return42 <markus.heiser@darmarit.de>
2023-04-14 07:07:48 +00:00
Markus Heiser 8c83547683 [mod] limiter: block unmaintained Farside instances
Since [bb3a01f8] has been merged to the Farside project, Farside instances do no
longer need to send requests to SearXNG instances [1].

There are some old unmaintained Farside instances on the web that continue to
query SearXNG instances --> we can safely block their requests.

[1] https://github.com/benbusby/farside/issues/95
[bb3a01f8] https://github.com/benbusby/farside/commit/bb3a01f8

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-13 16:53:37 +02:00
Markus Heiser c678b82625 [build] /static 2023-04-10 16:26:41 +02:00
Markus Heiser 962b4c719f [mod] Update input when selecting by TAB
When the user press [TAB] the input form should be filled with the highlighted
item from the autocomplete list, but not release a search / with other words:
what we now have by pressing once on [ENTER] should be mapped to the [TAB] key
and pressing [ENTER] once should release a search query. [1]

[1] https://github.com/searxng/searxng/issues/778#issuecomment-1016593816

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-10 16:26:41 +02:00
rinagorsha e9fb9f2705 [mod] Update input when selecting autocomplete prediction with keyboard
- Update input when selecting autocomplete prediction with keyboard
- Search immediately by pressing enter key
- Search immediately by clicking on an autocomplete suggestion

Related:
- https://github.com/searxng/searxng/issues/778
2023-04-10 14:36:05 +02:00
Markus Heiser 7d17d37ac7 [fix] don't show a category if there is no active engine in
When deactivate all the engines of a category, this category should disappeare.
This feature has been lost in commit 8e9ad1cc.

For better readability, webapp.get_enabled_categories() has been rewritten with
identical functionality.

Related:

- https://github.com/searxng/searxng/issues/1020
- https://github.com/searxng/searxng/issues/1604

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-10 09:22:21 +02:00
Markus Heiser fed2bab65b
Merge pull request #2329 from return42/fix-bing-no-descr
[fix] Bing-WEB: use <span class='algoSlug_icon'> for the description
2023-04-08 17:03:34 +02:00
Markus Heiser 0def555869 [build] /static 2023-04-08 11:10:34 +02:00
Markus Heiser f117f969d8 [mod] in the preference page, show !bang of subgrouping categories
The names of the subgrouping categories in the preference page are translated,
to use this categories the user needs to know by which !bang the category can be
selected.  Related to "Make 'non tab category' bangs discoverable" in [#690].

Related:

- [#690] https://github.com/searxng/searxng/issues/690
- https://github.com/searxng/searxng/issues/1604
- https://github.com/searxng/searxng/pull/1545

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-08 11:10:14 +02:00
Markus Heiser bbb2af7d94 [fix] minor typo in de/search-syntax page
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-08 10:25:52 +02:00
Markus Heiser 23ac964e35 [fix] Bing-WEB: use <span class='algoSlug_icon'> for the description
On some result items from Bing-WEB the `<span class='algoSlug_icon'>` tag is the
only tag that contains a description.  The issue can be reproduced by [1]::

    !bi vmware

[1] https://github.com/searxng/searxng/issues/1764#issuecomment-1417990531

Reported-by: @AlyoshaVasilieva
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-08 09:43:04 +02:00
Markus Heiser 393e14965a
Merge pull request #2326 from return42/ungrouped
[mod] clarify the difference of the default category and subgrouping
2023-04-07 13:34:24 +02:00
Markus Heiser 8f79dd7659 [doc] additional descriptions about categories & categories_as_tabs
Add missing documentation of PR [#634].  Related to checkbox "Document how to
categorize engines" in [#690].

Related:

- [#634] https://github.com/searxng/searxng/pull/634#issuecomment-1004757502
- [#690] https://github.com/searxng/searxng/issues/690
- https://github.com/searxng/searxng/issues/1604
- https://github.com/searxng/searxng/pull/1545

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-07 12:56:45 +02:00
Markus Heiser 2ffd446e5c [mod] clarify the difference of the default category and subgrouping
This PR does no functional change it is just an attempt to make more clear in
the code, what a default category is and what a subcategory is.  The previous
name 'others' leads to confusion with the **category 'other'**.

If a engine is not assigned to a category, the default is assigned::

    DEFAULT_CATEGORY = 'other'

If an engine has only one category and this category is shown as tab in the user
interface, this engine has no further subgrouping::

    NO_SUBGROUPING = 'without further subgrouping'

Related:

- https://github.com/searxng/searxng/issues/1604
- https://github.com/searxng/searxng/pull/1545

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-07 11:03:25 +02:00
searxng-bot 7850c77c2c [translations] update from Weblate
d2fb466c - 2023-04-02 - return42 <markus.heiser@darmarit.de>
5576f597 - 2023-04-02 - return42 <markus.heiser@darmarit.de>
4b28cab9 - 2023-03-31 - Vistaus <vistausss@fastmail.com>
2023-04-07 07:07:51 +00:00
Markus Heiser f46d0584ef
Merge pull request #2322 from return42/fix-2321
[fix] Gigablast.com has been erased
2023-04-06 23:49:58 +02:00
Markus Heiser 64e221426a
Merge pull request #2312 from return42/fix-1020-part-2
[fix] categories can't be removed from UI (categories_as_tabs)
2023-04-06 10:36:51 +02:00
Markus Heiser 5234e45010 [fix] Gigablast.com has been erased
[1] https://www.reddit.com/r/searchengines/comments/128wdcp/gigablastcom_has_been_erased/

Closes: https://github.com/searxng/searxng/issues/2321
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-06 08:22:57 +02:00
Markus Heiser 03f94962b6 [fix] limiter: never block a /healthz request
Related: https://github.com/searxng/searxng/issues/2310#issuecomment-1494417531
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-03 19:36:28 +02:00
Markus Heiser 0c4003ab2d [fix] categories can't be removed from UI (categories_as_tabs)
When using ``use_default_settings: true``, removing default categories from
settings.yml will not remove them from the UI.

The value ``categories_as_tabs`` is a dictionary type (a4c2cfb) and dictionary
types are merged additive by ``settings_loader.update_settings()``.

This patch replaces the default ``categories_as_tabs`` by the one from the
``user_settings``.

Related: https://github.com/searxng/searxng/issues/1019#issuecomment-1193145654
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-03 19:08:27 +02:00
Markus Heiser a762172bf7 [fix] engine ddg: quote !bangs in a request send to ddg
Closes: https://github.com/searxng/searxng/issues/392
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-03 09:52:16 +02:00
Markus Heiser 0430662189 [fix] engine google-News: fix decoding of URLs (part 2)
Follow up of 8de8070ed to fix the issue reported by AlyoshaVasilieva [1].

[1] https://github.com/searxng/searxng/issues/1959#issuecomment-1493300574

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-02 19:19:59 +02:00
Markus Heiser a5155a32c0
Merge pull request #2306 from return42/fix-1959
[fix] engine google-News: fix decoding of URLs
2023-04-02 08:02:37 +02:00
Markus Heiser 66810ce711 [mod] limiter: minor improvements
- requests without HTTP header 'Connection' or missing 'User-Agent' will be
  blocked by the limiter

- re_bot is related to 'User-Agent' and has been renamed to block_user_agent

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-01 19:42:49 +02:00
Markus Heiser 8de8070ed9 [fix] engine google-News: fix decoding of URLs
Google-News returns internal links where the origin URL is encoded in a
base64 (RFC 2045 aka URL-safe) string.

Closes: https://github.com/searxng/searxng/issues/1959
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-01 19:33:13 +02:00
Markus Heiser afd8fcce36 [mod] plugin limiter: improve the log messages
In debug mode more detailed logging is needed to evaluate if an access should
have been blocked by the limiter.

BTW: remove duplicate code checking bot signature ``re_bot.match(user_agent)``

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-01 09:20:58 +02:00
Markus Heiser 509afbbb84 [fix] engine seznam: fix issues reported by black & pylint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-31 17:25:39 +02:00
Venca24 c8d78355ff [fix] engine seznam 2023-03-31 16:11:27 +02:00
dependabot[bot] b99e028ed0
Bump sharp from 0.31.3 to 0.32.0 in /searx/static/themes/simple
Bumps [sharp](https://github.com/lovell/sharp) from 0.31.3 to 0.32.0.
- [Release notes](https://github.com/lovell/sharp/releases)
- [Changelog](https://github.com/lovell/sharp/blob/main/docs/changelog.md)
- [Commits](https://github.com/lovell/sharp/compare/v0.31.3...v0.32.0)

---
updated-dependencies:
- dependency-name: sharp
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-31 11:39:17 +00:00
Markus Heiser 68703ba22e
Merge pull request #2282 from searxng/dependabot/npm_and_yarn/searx/static/themes/simple/master/ionicons-7.1.0
Bump ionicons from 6.1.3 to 7.1.0 in /searx/static/themes/simple
2023-03-31 13:36:12 +02:00
searxng-bot c1c24fc231 [translations] update from Weblate
17ad1118 - 2023-03-29 - return42 <markus.heiser@darmarit.de>
61446791 - 2023-03-29 - return42 <markus.heiser@darmarit.de>
2023-03-31 07:07:59 +00:00
Markus Heiser 270ad18897 [fix] engine flickr: adapt to the new data model from flicker's response
Closes: https://github.com/searxng/searxng/issues/1879
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-30 21:04:53 +02:00
Markus Heiser 2b8dfab33f [fix] engine gigablast: add &userid=<User ID>&code=<Feed Code>
Gigablast's API does block unauthorized request[1].

[1] https://gigablast.com/searchfeed.html

Closes: https://github.com/searxng/searxng/issues/1454
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-29 16:18:02 +02:00
Markus Heiser d179b62cf5
Merge pull request #2288 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2023-03-29 13:48:32 +02:00
Markus Heiser 381c6751d6
Merge pull request #2289 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2023-03-29 13:48:03 +02:00
Markus Heiser fc51d9a0fe
Merge pull request #2291 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2023-03-29 13:46:40 +02:00
Markus Heiser 2fbe4ab0c0
Merge pull request #2292 from searxng/update_data_update_engine_descriptions.py
Update searx.data - update_engine_descriptions.py
2023-03-29 13:46:14 +02:00
dalf 4c80340b62 Update searx.data - update_engine_descriptions.py 2023-03-29 00:28:45 +00:00
dalf b39ce7ff82 Update searx.data - update_currencies.py 2023-03-29 00:16:21 +00:00
dalf 814ac8cacb Update searx.data - update_ahmia_blacklist.py 2023-03-29 00:16:17 +00:00
dalf 43d30cab81 Update searx.data - update_firefox_version.py 2023-03-29 00:16:14 +00:00
Markus Heiser 6f9e678346 [fix] engine: google has changed the layout of its response
Since 28. March google has changed its response, this patch fixes the google
engine to scrap out the results & images from the new designed response.

closes: https://github.com/searxng/searxng/issues/2287

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-28 14:39:16 +02:00
dalf 1498202b0b Update searx.data - update_engine_traits.py 2023-03-24 11:30:18 +01:00
Markus Heiser 16f0db4493 [mod] replace utils.match_language by locales.match_locale
This patch replaces the *full of magic* ``utils.match_language`` function by a
``locales.match_locale``.  The ``locales.match_locale`` function is based on the
``locales.build_engine_locales`` introduced in 9ae409a0 [1].

In the past SearXNG did only support a search by a language but not in a region.
This has been changed a long time ago and regions have been added to SearXNG
core but not to the engines.  The ``utils.match_language`` was the function to
handle the different aspects of language/regions in SearXNG core and the
supported *languages* in the engine.  The ``utils.match_language`` did it with
some magic and works good for most use cases but fails in some edge case.

To replace the concurrence of languages and regions in the SearXNG core the
``locales.build_engine_locales`` was introduced in 9ae409a0 [1].  With the last
patches all engines has been migrated to a ``fetch_traits`` and a
language/region concept that is based on ``locales.build_engine_locales``.

To summarize: there is no longer a need for the ``locales.match_language``.

[1] https://github.com/searxng/searxng/pull/1652

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 4d4aa13e1f [mod] remove obsolete EngineTraits.supported_languages
All engines has been migrated from ``supported_languages`` to the
``fetch_traits`` concept.  There is no longer a need for the obsolete code that
implements the ``supported_languages`` concept.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 96a2eec3b5 [mod] Archlinux Wiki: improved request API & upgrade to data_type: traits_v1
re-implementation of the Archlinux Wiki:

- fetch_traits(): fetch languages, wiki URLs and title arguments
- add content field to the result list
- add documentation

Wikis from wiki.archlinux.fr, wiki.archlinux.ro, archtr.org/wiki do no longer
exists (has been merged in the main wiki).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 057e9bc1d1 [mod] SepiaSearch: re-engineered & upgrade to data_type: traits_v1
- fetch_traits() SepiaSearch and Peertube are using identical languages.
  Replace module's dictionary `supported_languages` by `engine.traits.languages`
  (data_type: `traits_v1`).
- fixed code to pass pylint
- request(): add argument boostLanguages
- response(): is replaced by peertube's video_response() function, which adds
  metadata from channel name, host & tags

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 8a8c584fec [mod] Dailymotion: improved request API & upgrade to data_type: traits_v1
- fetch_traits(): fetch locales (and languages) from dailymotion API
- removed obsolete data-type "supported_languages"
- add documentation
- improved argument list of the HTTP request:
  - add argument: family_filter_map
  - add conditional argument: localization
    Don't add localization and country arguments if the user does select a
    language (:de, :en, ..)
- improve code quality (mainly improve readability)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 2499899554 [mod] Google: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the Google engines including a improved language
and region handling based on the engine.traits_v1 data.

When ever possible the implementations of the Google engines try to make use of
the async REST APIs.  The get_lang_info() has been generalized to a
get_google_info() function / especially the region handling has been improved by
adding the cr parameter.

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - Google (WEB),
  - Google images,
  - Google news,
  - Google scholar and
  - Google videos

  and remove data from obsolete data type "supported_languages".

  A traits.custom type that maps region codes to *supported_domains* is fetched
  from https://www.google.com/supported_domains

searx/autocomplete.py:
  Reversed engineered autocomplete from Google WEB.  Supports Google's languages and
  subdomains.  The old API suggestqueries.google.com/complete has been replaced
  by the async REST API: https://{subdomain}/complete/search?{args}

searx/engines/google.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - always use the async REST API (formally known as 'use_mobile_ui')
  - use *supported_domains* from traits
  - improved the result list by fetching './/div[@data-content-feature]'
    and parsing the type of the various *content features* --> thumbnails are
    added

searx/engines/google_images.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - if exists, freshness_date is added to the result
  - issue 1864: result list has been improved a lot (due to the new cr parameter)

searx/engines/google_news.py
  Reverse engineering and extensive testing ..
  - fetch_traits():  Fetch languages & regions from Google properties.
    *supported_domains* is not needed but a ceid list has been added.
  - different region handling compared to Google WEB
  - fixed for various languages & regions (due to the new ceid parameter) /
    avoid CONSENT page
  - Google News do no longer support time range
  - result list has been fixed: XPath of pub_date and pub_origin

searx/engines/google_videos.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - add paging support
  - implement a async request ('asearch': 'arc' & 'async':
    'use_ac:true,_fmt:html')
  - simplified code (thanks to '_fmt:html' request)
  - issue 1359: fixed xpath of video length data

searx/engines/google_scholar.py
  - fetch_traits():  Fetch languages & regions from Google properties.
  - use *supported_domains* from traits
  - request(): include patents & citations
  - response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager)
  - hardening XPath to iterate over results
  - fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class)
  - issue 1769 fixed: new request implementation is no longer incompatible

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser c80e82a855 [mod] DuckDuckGo: reversed engineered & upgrade to data_type: traits_v1
Partial reverse engineering of the DuckDuckGo (DDG) engines including a
improved language and region handling based on the enigne.traits_v1 data.

- DDG Lite
- DDG Instant Answer API
- DDG Images
- DDG Weather

docs/src/searx.engine.duckduckgo.rst:
  Online documentation of the DDG engines (make docs.live)

searx/data/engine_traits.json
  Add data type "traits_v1" generated by the fetch_traits() functions from:

  - "duckduckgo" (WEB),
  - "duckduckgo images" and
  - "duckduckgo weather"

  and remove data from obsolete data type "supported_languages".

searx/autocomplete.py:
  Reversed engineered Autocomplete from DDG.  Supports DDG's languages.

searx/engines/duckduckgo.py:
  - fetch_traits():  Fetch languages & regions from DDG.

  - get_ddg_lang(): Get DDG's language identifier from SearXNG's locale.  DDG
    defines its languages by region codes.  DDG-Lite does not offer a language
    selection to the user, only a region can be selected by the user.

  - Cache ``vqd`` value: The vqd value depends on the query string and is needed
    for the follow up pages or the images loaded by a XMLHttpRequest (DDG
    images).  The ``vqd`` value of a search term is stored for 10min in the
    redis DB.

  - DDG Lite engine: reversed engineered request method with improved Language
    and region support and better ``vqd`` handling.

searx/engines/duckduckgo_definitions.py: DDG Instant Answer API
  The *instant answers* API does not support languages, or at least we could not
  find out how language support should work.  It seems that most of the features
  are based on English terms.

searx/engines/duckduckgo_images.py: DDG Images
  Reversed engineered request method.  Improved language and region handling
  based on cookies and the enigne.traits_v1 data.  Response: add image format to
  the result list

searx/engines/duckduckgo_weather.py: DDG Weather
  Improved language and region handling based on cookies and the
  enigne.traits_v1 data.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser e9afc4f8ce [mod] Startpage: reversed engineered & upgrade to data_type: traits_v1
One reason for the often seen CAPTCHA of the Startpage requests are the
incomplete requests SearXNG sends to startpage.com: this patch is a complete new
implementation of the ``request()`` function, reversed engineered from the
Startpage's search form.  The new implementation:

- use traits of data_type: traits_v1 and drop deprecated data_type: supported_languages
- adds time-range support
- adds save-search support
- fix searxng/searxng/issues 1884
- fix searxng/searxng/issues 1081 --> improvements to avoid CAPTCHA

In preparation for more categories (News, Images, Videos ..) from Startpage, the
variable ``startpage_categ`` was set up.  The default value is ``web`` and other
categories from Startpage are not yet implemented.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 858aa3e604 [mod] wikipedia & wikidata: upgrade to data_type: traits_v1
BTW this fix an issue in wikipedia: SearXNG's locales zh-TW and zh-HK are now
using language `zh-classical` from wikipedia (and not `zh`).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser e0a6ca96cc [doc] add a description of bing engines (web, news, video, images)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 15eaf0f15f [mod] bing_news: use async API & upgrade to data_type: traits_v1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser ff80e7637e [mod] bing_images: use async API & upgrade to data_type: traits_v1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser bc21d28298 [mod] bing_videos: use async API & upgrade to data_type: traits_v1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser d0f465e6fa [mod] bing: add time_range support & upgrade to data_type: traits_v1
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser c9cd376186 [mod] replace searx.languages by searx.sxng_locales
With the language and region tags from the EngineTraitsMap the handling of
SearXNG's tags of languages and regions has been normalized and is no longer
a *mystery*.  The "languages" became "locales" that are supported by babel and
by this, the update_engine_traits.py can be simplified a lot.

Other code places can be simplified as well, but these simplifications
should (respectively can) only be done when none of the engines work with the
deprecated EngineTraits.supported_languages interface anymore.

This commit replaces searx.languages by searx.sxng_locales and fix the naming of
some names from "language" to "locale" (e.g. language_codes --> sxng_locales).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 7daf4f95ef [mod] Wikipedia: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Wikipedia engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser f78f908383 [mod] Google: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Google engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser dba8977b09 [mod] DuckDuckGo: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the DuckDuckGo engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser ef143729a0 [mod] yahoo: fetch engine traits (data_type: traits_v1)
Implements a fetch_traits function for the Yahoo engine.

.. note::

   Includes migration of the request methode from 'supported_languages' to
   'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser c1ae2ef57c [mod] qwant: fetch engine traits (data_type: traits_v1)
Implements a fetch_traits function for the Qwant engines.

.. note::

   Includes migration of the request methode from 'supported_languages' to
   'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser fc0c775030 [mod] Dailymotion: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Dailymotion engine.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 61383edb27 [mod] Startpage: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Startpage engine.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser d3aa690a7a [mod] bing: fetch engine traits (data_type: supported_languages)
Implements a fetch_traits function for the Bing engines.

.. note::

   Does not include migration of the request methode from 'supported_languages'
   to 'traits' (EngineTraits) object!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser a7fe22770a [mod] Peertube: re-engineered & upgrade to data_type: traits_v1
- fetch_traits(): Fetch languages from peertube's search-index source code.

  [mod] Include migration of the request methode from 'supported_languages'
        to 'traits' (EngineTraits) object.
  [fix] old supported_languages_url is no longer valid since the sources
        has been moved to a different path.

- fixed code to pass pylint
- request(): complete re-implementation based on the API docs [1]
- response(): complete re-implementation, adds serveral fields missed before
- add source code documentation

[1] https://docs.joinpeertube.org/api-rest-reference.html#tag/Search/operation/searchVideos

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
Markus Heiser 6e5f22e558 [mod] replace engines_languages.json by engines_traits.json
Implementations of the *traits* of the engines.

Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*.  Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.

To load traits from the persistence::

    searx.enginelib.traits.EngineTraitsMap.from_data()

For new traits new properties can be added to the class::

    searx.enginelib.traits.EngineTraits

.. hint::

   Implementation is downward compatible to the deprecated *supported_languages
   method* from the vintage implementation.

   The vintage code is tagged as *deprecated* an can be removed when all engines
   has been ported to the *traits method*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24 10:37:42 +01:00
dependabot[bot] 912f877dec
Bump ionicons from 6.1.3 to 7.1.0 in /searx/static/themes/simple
Bumps [ionicons](https://github.com/ionic-team/ionicons) from 6.1.3 to 7.1.0.
- [Release notes](https://github.com/ionic-team/ionicons/releases)
- [Changelog](https://github.com/ionic-team/ionicons/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ionic-team/ionicons/compare/v6.1.3...v7.1.0)

---
updated-dependencies:
- dependency-name: ionicons
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-03-24 07:57:14 +00:00
searxng-bot 9f3a57c901 [translations] update from Weblate
abfec8f4 - 2023-03-23 - return42 <markus.heiser@darmarit.de>
f02ea21c - 2023-03-23 - return42 <markus.heiser@darmarit.de>
3fc6c653 - 2023-03-20 - chenghui-lee <chlee9926@gmail.com>
342bbf46 - 2023-03-20 - return42 <markus.heiser@darmarit.de>
2023-03-24 07:07:52 +00:00
Solirs ac169a0f75 Pass black formatting test 2023-03-21 00:41:36 +01:00
Solirs e26bce33d4 WIKIDATA: Add description for results 2023-03-21 00:14:54 +01:00
Markus Heiser b61b845951
Merge pull request #2266 from return42/shuffle-cipher
[mod] Shuffle httpx's default ciphers of a SSL context randomly.
2023-03-20 12:28:05 +01:00
Markus Heiser 94430e104c
Merge pull request #2238 from return42/fix-2027
[fix] fix threshold in replace_auto_language
2023-03-19 15:30:37 +01:00
Markus Heiser f2962a2f4a
Merge pull request #2239 from return42/fix-eslintrc
[fix] remove duplicate key in simple theme ESLint configuration
2023-03-19 15:30:12 +01:00
Markus Heiser 8fa54ffddf [mod] Shuffle httpx's default ciphers of a SSL context randomly.
From the analyse of @9Ninety [1] we know that DDG (and may be other engines / I
have startpage in mind) does some kind of TLS fingerprint to block bots.

This patch shuffles the default ciphers from httpx to avoid a cipher profile
that is known to httpx (and blocked by DDG).

[1] https://github.com/searxng/searxng/issues/2246#issuecomment-1467895556

----

From `What Is TLS Fingerprint and How to Bypass It`_

> When implementing TLS fingerprinting, servers can't operate based on a
> locked-in whitelist database of fingerprints.  New fingerprints appear
> when web clients or TLS libraries release new versions. So, they have to
> live off a blocklist database instead.
> ...
> It's safe to leave the first three as is but shuffle the remaining ciphers
> and you can bypass the TLS fingerprint check.

.. _What Is TLS Fingerprint and How to Bypass It:
   https://www.zenrows.com/blog/what-is-tls-fingerprint#how-to-bypass-tls-fingerprinting

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Closes: https://github.com/searxng/searxng/issues/2246
2023-03-19 13:40:31 +01:00
Markus Heiser 677903c355
Merge pull request #2257 from Solirs/fix_bad_escape
re.escape() the query in highlight_content to prevent a server side error.
2023-03-17 08:54:54 +01:00
Solirs fbb0e9d275 [fix] server side error: escape backslashes in the query highlight_content
Any backslash escapes in the replacement are processed [1], backslashes should
be escaped [2].

[1] https://docs.python.org/3/library/re.html#re.sub
[2] https://docs.python.org/3/library/re.html#re.escape

closes:
- https://github.com/searxng/searxng/issues/2256
- https://github.com/searxng/searxng/issues/2250
2023-03-17 08:46:00 +01:00
searxng-bot 86c3757872 [translations] update from Weblate
32926a19 - 2023-03-15 - return42 <markus.heiser@darmarit.de>
7aabc876 - 2023-03-16 - Linerly <linerly@protonmail.com>
c0ed00f5 - 2023-03-14 - SonoAX <giovanniilgiovo@gmail.com>
6cf287f6 - 2023-03-13 - RhysJones <proladrhys123@outlook.com>
8c4c5f83 - 2023-03-12 - Cavemanly <k.adel.2m@protonmail.com>
dffe61fa - 2023-03-10 - return42 <markus.heiser@darmarit.de>
c7736cac - 2023-03-10 - BalkanMadman <zurabid2016@gmail.com>
e831b8e3 - 2023-03-10 - BalkanMadman <zurabid2016@gmail.com>
ef3c60af - 2023-03-10 - return42 <markus.heiser@darmarit.de>
c046a677 - 2023-03-07 - BalkanMadman <zurabid2016@gmail.com>
142041d6 - 2023-03-05 - return42 <markus.heiser@darmarit.de>
119b51df - 2023-03-05 - return42 <markus.heiser@darmarit.de>
2023-03-17 07:07:53 +00:00
Alexandre Flament 3e9cddc606
rollback test 2023-03-15 19:55:20 +01:00
Alexandre Flament 41ed0ef0c7
test 2023-03-15 19:53:53 +01:00
Markus Heiser 097d092a7f
Merge pull request #2224 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2023-03-15 18:36:29 +01:00
Markus Heiser 85ef1af343
Merge pull request #2222 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2023-03-15 18:35:36 +01:00
Markus Heiser a7f1649190 [fix] remove duplicate key in simple theme ESLint configuration
Partial merge of [PR-1736]

[PR-1736] https://github.com/searxng/searxng/pull/1736

Suggested-by: @FunctionalHacker in [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-05 09:12:13 +01:00
Markus Heiser 150a90c84e [fix] fix threshold in replace_auto_language
[1] https://github.com/searxng/searxng/pull/2027#pullrequestreview-1322157677
[2] https://github.com/searxng/searxng/pull/1969#issuecomment-1345354529

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-05 08:29:58 +01:00
searxng-bot 1f36fc3a45 [translations] update from Weblate
0d8ebfe1 - 2023-03-02 - AHOHNMYC <lqwh2h2cwa@protonmail.com>
1358dd6d - 2023-03-03 - mystery-z <07juwonc@kakao.com>
4d8c13db - 2023-03-01 - lhostfree951 <freeehost9191@gmail.com>
1ae581b6 - 2023-02-28 - tygyh <jonis9898@hotmail.com>
0003698f - 2023-02-28 - ewm <gnu.ewm@protonmail.com>
31c79617 - 2023-02-28 - gjveld <gjveld@gmail.com>
9015ec73 - 2023-02-28 - gallegonovato <fran-carro@hotmail.es>
03619a68 - 2023-02-25 - BalkanMadman <zurabid2016@gmail.com>
fa90585b - 2023-02-25 - BalkanMadman <zurabid2016@gmail.com>
c902c5e5 - 2023-02-26 - tentsbet <remendne@pentrens.jp>
2023-03-03 07:08:08 +00:00
Alexandre Flament 714e83d5ea
Merge pull request #2220 from Solirs/gentoo_engine_timeout
Increase timeout for gentoo wiki engine
2023-03-01 17:57:21 +01:00
Alexandre Flament 1632f18631
Merge pull request #2227 from searxng/update_data_update_engine_descriptions.py
Update searx.data - update_engine_descriptions.py
2023-03-01 17:52:57 +01:00
Alexandre Flament 5bbbb14b62
Merge pull request #2226 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2023-03-01 17:52:02 +01:00
dalf 5042d94dea Update searx.data - update_engine_descriptions.py 2023-03-01 01:48:22 +00:00
dalf e30a45812f Update searx.data - update_ahmia_blacklist.py 2023-03-01 01:38:07 +00:00
dalf 935415bfcf Update searx.data - update_currencies.py 2023-03-01 01:37:46 +00:00
dalf ccd00518fd Update searx.data - update_firefox_version.py 2023-03-01 01:37:42 +00:00
dalf d7f10909fa Update searx.data - update_wikidata_units.py 2023-03-01 01:37:25 +00:00
Solirs 35fbb3578b Increase timeout for gentoo wiki engine 2023-02-28 13:54:44 +01:00
Alexandre Flament d669da81fb
Merge pull request #2027 from dalf/fix_2018
Add "auto" as a language.
2023-02-20 12:17:38 +01:00
searxng-bot 297e463e49 [translations] update from Weblate
8ff0fa33 - 2023-02-19 - return42 <markus.heiser@darmarit.de>
2023-02-19 11:46:59 +00:00
Markus Heiser 0b1444b61e [doc] improved docs of implementations for automatic speech recognition
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-19 10:09:52 +00:00
Markus Heiser 363203c579
Merge pull request #2201 from return42/fix-2190
[doc] slight improvements to the doc of the settings (base_url)
2023-02-18 18:21:14 +01:00
Alexandre Flament 6748e8e2d5 Add "Auto-detected" as a language.
When the user choose "Auto-detected", the choice remains on the following queries.
The detected language is displayed.

For example "Auto-detected (en)":
* the next query language is going to be auto detected
* for the current query, the detected language is English.

This replace the autodetect_search_language plugin.
2023-02-17 15:17:36 +00:00
Markus Heiser bb83036f48 [fix] typo in searx/plugins/tor_check.py
Related: https://github.com/searxng/searxng/pull/2189

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-17 13:09:14 +01:00
Markus Heiser 5820dc78ce [doc] slight improvements to the doc of the settings (base_url)
Closes: https://github.com/searxng/searxng/issues/2190

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-17 12:08:58 +01:00
Markus Heiser 52f6bc745b
Merge pull request #2188 from ahmad-alkadri/fix/petalsearch
Fix the petalsearch engine
2023-02-15 13:57:28 +01:00
Markus Heiser ba5b6c382f
Merge pull request #2192 from return42/fix-tineye
[mod] disbale engine tineye by default
2023-02-15 06:43:41 +01:00
Ahmad Alkadri f6af59899b Fix petalsearch and remove petalsearch news 2023-02-14 18:43:55 +01:00
Allan Nordhøy 2be373a18f [fix] spelling: Tor, SearXNG 2023-02-14 17:13:53 +01:00
Markus Heiser 7d446dfdb2 [mod] disbale engine tineye by default
Tineye becomes active as soon as a https:// signature is found in the search
term, but most of the time a reverse image search is not requested when a URL is
specified, often the URL is just from a C&P.

The frequent requests to tineye lead in the end to the SearXNG instance being
blocked by tineye and the user seeing unexpected error messages.

BTW: many maintainers have disabled this engine in their local SearXNG settings.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-14 08:27:19 +01:00
Markus Heiser 3abff182ea [fix] remove engine neeva from settings.yml
Engine is broken and can't by used any longer as a simple XPath engine.
@allendema tested a engines/neeva.py version using json from the dom, but
without luck: There was some kind of captcha for pagination.

[1] https://github.com/searxng/searxng/issues/2007#issuecomment-1426061698

Closes: https://github.com/searxng/searxng/issues/2007
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 18:46:37 +01:00
Lea a530ac4099 fix: typo in suggestions 2023-02-10 15:33:25 +01:00
Markus Heiser 415f6398d6 [fix] time_range in HTML form should not be a string 'None'
Before this patch SearXNG returns 'time_range': 'None'::

    <form method="GET" action="/searx/search">
      ...
      <input type="hidden" name="time_range" value="None">
      ...
    </form>

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 15:32:21 +01:00
wibyweb 6707354bc8 [mod] engine wiby: add pagination
Suggested by: @wibyweb in searx https://github.com/searx/searx/pull/3465

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 15:31:24 +01:00
Markus Heiser 4c06837a50 [mod] make python code pylint 2.16.1 compliant
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-10 13:59:21 +01:00
Markus Heiser 7320b0c796
Merge pull request #2130 from searxng/translations_update
Update translations
2023-02-04 17:53:20 +01:00
dependabot[bot] a5a73d77b3
Bump grunt from 1.5.3 to 1.6.1 in /searx/static/themes/simple
Bumps [grunt](https://github.com/gruntjs/grunt) from 1.5.3 to 1.6.1.
- [Release notes](https://github.com/gruntjs/grunt/releases)
- [Changelog](https://github.com/gruntjs/grunt/blob/main/CHANGELOG)
- [Commits](https://github.com/gruntjs/grunt/compare/v1.5.3...v1.6.1)

---
updated-dependencies:
- dependency-name: grunt
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-04 08:10:18 +00:00
Markus Heiser 257dc7d6c4 [fix-2146] set different HTTP Referer header to DuckDuckGo requests
For what ever reasons, ddg-lite don't like the Referer

  https://lite.duckduckgo.com/

In an interactive session in the WEB browser the the Reverer has exactly this
value, but ddg-lite don't like this value when the request is build up by
SearXNG.  The new value is:

  https://google.com/

What fakes a user comes from a google link.

Related: https://github.com/searxng/searxng/pull/2081
Closes: https://github.com/searxng/searxng/issues/2146

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-02-03 08:45:51 +01:00
searxng-bot 8db35f8e9c [translations] update from Weblate
009d48c2 - 2023-02-01 - tygyh <jonis9898@hotmail.com>
a5353bad - 2023-02-01 - Mooo <moose@mail.ru>
1617f458 - 2023-01-28 - emirhan <xemirhankoc@gmail.com>
79bc60f4 - 2023-01-29 - ewm <gnu.ewm@protonmail.com>
2023-02-03 07:15:25 +00:00
Alexandre Flament 9d102fb08f
Merge pull request #2132 from dalf/update_pr_1967
search.suspended_time settings: bug fixes
2023-01-29 20:48:43 +01:00
Markus Heiser feccee01c0 [doc] Add doc-strings to searx.exceptions
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-01-29 19:06:19 +01:00
Markus Heiser 4f808088f1
Merge pull request #2139 from dalf/fix_update_languages
Fix update languages
2023-01-29 17:59:59 +01:00
Markus Heiser d989c9429f
Merge pull request #2134 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2023-01-29 17:59:40 +01:00
Markus Heiser 5c2667b956
Merge pull request #2135 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2023-01-29 17:59:03 +01:00
Markus Heiser e6a17f2fae
Merge pull request #2136 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2023-01-29 17:58:22 +01:00
Markus Heiser c4cbf8829a
Merge pull request #2137 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2023-01-29 17:57:56 +01:00
Alexandre Flament bfca63c536 wikipedia engine: update _fetch_supported_languages
the layout https://meta.wikimedia.org/wiki/List_of_Wikipedias has changed
2023-01-29 10:01:58 +00:00
Alexandre Flament 8256de2fe8 peertube engine: update _fetch_supported_languages
There is now an API to get the list of supported languages
https://docs.joinpeertube.org/api-rest-reference.html#tag/Video/operation/getLanguages
2023-01-29 10:01:54 +00:00
dalf f900e33092 Update searx.data - update_engine_descriptions.py 2023-01-29 01:33:29 +00:00
dalf dafe026815 Update searx.data - update_currencies.py 2023-01-29 01:24:14 +00:00
dalf 47435be70b Update searx.data - update_firefox_version.py 2023-01-29 01:23:43 +00:00
dalf 53a4a90d0c Update searx.data - update_wikidata_units.py 2023-01-29 01:23:42 +00:00
dalf 9dd95457a9 Update searx.data - update_ahmia_blacklist.py 2023-01-29 01:23:42 +00:00
Alexandre Flament 37addec69e search.suspended_time settings: bug fixes
* fix type in settings.yml: replace suspend_times by suspended_times
* always use delay defined in settings.yml:
  * HTTP status 402 and 403: read the value from settings.yml instead of using the hardcoded value of 1 day.
  * startpage engine: CAPTCHA suspend the engine for one day instead of one week
2023-01-28 10:24:14 +00:00
Alexandre Flament 28356f4284
Merge pull request #2115 from searxng/dependabot/pip/master/flask-babel-3.0.0
Bump flask-babel from 2.0.0 to 3.0.0
2023-01-27 10:43:17 +01:00
mrpaulblack b3883dc395 [build] /static 2023-01-23 23:00:38 +01:00
mrpaulblack 8ead23347f [fix] use input elements for pagination requests
* use html input elements instead of buttons for the pagination forms at the bottom of the result page
* move the less section that hides the pagination number widget on mobile to the mobile section
* clean up the less code for styling the numbers of the pagination widget
* fix: add the margin for box results (like in news category) to the bottom of the result to have a margin between pagination widget and article result
2023-01-23 22:56:55 +01:00
llmII d2b2300ee1 Feature: Pagination Widget with list of numbers.
Adds to the navigation widget, preserving forward/backward nav, and
inserting a list of clickable page numbers between them.

Phone sized devices continue without this widget as deterministic
display under small screen sizes has not been solved.

The widget is agnostic to the actual amount of pages there are that one
can navigate to and as such shows all plausible, albeit not necessarilly
valid, possibilities.

This widget does not interfere with infinite scroll in any fashion.
2023-01-23 21:22:06 +01:00
Alexandre Flament 06b5068a2e SearXNG version: YYYY.MM.DD without leading zero 2023-01-22 10:45:00 +00:00
Alexandre Flament a9d6f7532a weblate: migration to https://translate.codeberg.org/ 2023-01-21 15:45:12 +00:00
searxng-bot 6e4c8b3516 [translations] update from Weblate 2023-01-20 11:32:23 +00:00
Alexandre Flament f759a84af4 SearXNG version: fix make docker
continuation of #2117
related to #2111

This commit:
* fixes the Docker tag using an additional variable DOCKER_TAG, see searx/version.py
* fixes the Docker labels org.label-schema.vcs-ref and org.opencontainers.image.revision
* adds searx/version_frozen to .gitignore
2023-01-20 10:46:00 +00:00
dependabot[bot] 70fff2ebf1 Bump flask-babel from 2.0.0 to 3.0.0
Bumps [flask-babel](https://github.com/python-babel/flask-babel) from 2.0.0 to 3.0.0.
- [Release notes](https://github.com/python-babel/flask-babel/releases)
- [Changelog](https://github.com/python-babel/flask-babel/blob/master/CHANGELOG)
- [Commits](https://github.com/python-babel/flask-babel/compare/v2.0.0...v3.0.0)

---
updated-dependencies:
- dependency-name: flask-babel
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-20 09:38:30 +00:00
Alexandre Flament 0515f2974b Version number follows PEP440 : replace - by +
Close #2111
2023-01-20 09:05:44 +00:00
Alexandre Flament 6d72ef3cbe
Merge pull request #2109 from ahmad-alkadri/fix/highlight-full-word
Standalone words highlighting for query result in non-CJK characters
2023-01-17 23:24:04 +01:00
Alexandre Flament 13b0c251c4
Merge pull request #2100 from nexryai/master
Add goo engine
2023-01-15 23:08:28 +01:00
ahmad-alkadri 99b5272d9a A little fix and modified the testing for content highlight 2023-01-15 16:51:31 +01:00
Léon Tiekötter 0cedb1c6d8 Add search.suspended_times settings
Make suspended_time changeable in settings.yml
Allow different values to be set for different exceptions.

Co-authored-by: Alexandre Flament <alex@al-f.net>
2023-01-15 09:00:32 +00:00
ahmad-alkadri 6c421110b5 Perso branch - added cjk check and enclosement 2023-01-14 23:00:08 +00:00
searxng-bot 3e5ea08504 [translations] update from Weblate
6a57ee0b - 2023-01-11 - Markus Heiser <markus.heiser@darmarit.de>
b180a810 - 2023-01-06 - Nik Drešar <nik.dresar@gmail.com>
8710174a - 2023-01-11 - Markus Heiser <markus.heiser@darmarit.de>
529a2591 - 2023-01-08 - Markus Heiser <markus.heiser@darmarit.de>
a254bd01 - 2023-01-08 - Markus Heiser <markus.heiser@darmarit.de>
860c3fce - 2023-01-11 - Markus Heiser <markus.heiser@darmarit.de>
c140fb52 - 2023-01-06 - Markus Heiser <markus.heiser@darmarit.de>
2023-01-13 07:14:55 +00:00
nexryai 4e7bb1bf9a
Add goo engine 2023-01-12 16:28:09 +09:00
Ahmad Alkadri 7fc8d72889 [fix] bing: parsing result; check to see if the element contains links
This patch is to hardening the parsing of the bing response:

1. To fix [2087] check if the selected result item contains a link, otherwise
   skip result item and continue in the result loop.  Increment the result
   pointer when a result has been added / the enumerate that counts for skipped
   items is no longer valid when result items are skipped.

   To test the bugfix use:   ``!bi :all cerbot``

2. Limit the XPath selection of result items to direct children nodes (list
   items ``li``) of the ordered list (``ol``).

   To test the selector use: ``!bi :en pontiac aztek wiki``

   .. in the result list you should find the wikipedia entry on top,
   compare [2068]

[2087] https://github.com/searxng/searxng/issues/2087
[2068] https://github.com/searxng/searxng/issues/2068
2023-01-09 15:08:24 +01:00
Milad-Laly cf4db4be37 [fix] Mojeek Xpath showing suggestions and searches + add lang support 2023-01-09 09:33:47 +01:00
Markus Heiser 54e63839db
Merge pull request #2088 from llmII/fix-suggestions
Fix #2083
2023-01-08 18:53:32 +01:00
Markus Heiser 9d6b048fba [build] /static 2023-01-08 18:39:37 +01:00
llmII 33d97fd01a Fix #2083
This changes the Suggestions to be a single column, not a wrapping row,
changing the input to be incapable of overflowing into visually adjacent
elements.
2023-01-08 10:44:58 -06:00
allixx 1407a055b2 [simple theme] set tab index for search page input field #1891
Previously it took five tab keypresses to reach search input field
on search results page. Now one tab press is enough.
2023-01-08 13:41:48 +03:00
ahmad-alkadri 9ee99423fe [fix] Bing-Web engine: XPath to get the wikipedia result
Modify the XPath selector to get the wikipedia result plus small fixes.

About result content: especially with the Wikipedia result, we'd get several
paragraph elements, only the first paragraph would be taken and displayed on the
search result
2023-01-08 09:11:16 +01:00
Markus Heiser 4e355564d2
Merge pull request #1903 from LalleSX/patch-1
Added info about adding the enginge to the brownser
2023-01-07 15:03:12 +01:00
PrivateLalle 212acf7f7b [mod] about page: add information about adding enginge to the brownser 2023-01-07 14:48:06 +01:00
Markus Heiser 4cd3be8964 [fix] indentation and line size of the the markdown files
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-01-07 14:46:31 +01:00
Markus Heiser b2a80347e3 [build] /static 2023-01-07 11:53:51 +01:00
ahmad-alkadri db70c262c3 Add flex-wrap to the list of engines 2023-01-07 11:53:05 +01:00
Rudis Muiznieks 128b8c7f0a
Add HTTP Referer header to DuckDuckGo requests
closes #2080
2023-01-06 16:07:37 -06:00
Markus Heiser b241015e93
Merge pull request #2078 from searxng/dependabot/pip/master/pygments-2.14.0
Bump pygments from 2.13.0 to 2.14.0
2023-01-06 08:40:46 +01:00
Markus Heiser e9a7a6aea6 [build] /static 2023-01-06 08:35:52 +01:00
searxng-bot 2420ab6fc3 [translations] update from Weblate
b1c60c5b - 2022-12-30 - wldkwl <gayisurdad@hotmail.com>
89133548 - 2023-01-05 - Linerly <linerly@protonmail.com>
4398061a - 2023-01-02 - Markus Heiser <markus.heiser@darmarit.de>
9857916a - 2023-01-04 - p6n7l <nichita@tutanota.com>
0ba50078 - 2023-01-04 - Markus Heiser <markus.heiser@darmarit.de>
0189e648 - 2023-01-02 - Markus Heiser <markus.heiser@darmarit.de>
24ded833 - 2023-01-01 - Markus Heiser <markus.heiser@darmarit.de>
2023-01-06 07:14:08 +00:00
searxng-bot 38c22c8c31 [translations] update from Weblate
ea56f7b6 - 2022-12-26 - Sean Botha <seanbotha@gmail.com>
0dbbb031 - 2022-12-23 - gkkulik <gregorykkulik@gmail.com>
81f33cb7 - 2022-12-28 - Markus Heiser <markus.heiser@darmarit.de>
5ff82460 - 2022-12-28 - peterT1D <kozlovic.peter@gmail.com>
0ea6d8c3 - 2022-12-28 - Markus Heiser <markus.heiser@darmarit.de>
b19bb597 - 2022-12-28 - Markus Heiser <markus.heiser@darmarit.de>
904fa9ab - 2022-12-24 - Markus Heiser <markus.heiser@darmarit.de>
e7c7d624 - 2022-12-29 - Droid <droidpy587@gmail.com>
2022-12-30 07:14:21 +00:00
dalf 0639769274 Update searx.data - update_firefox_version.py 2022-12-29 09:17:56 +01:00
Markus Heiser 174e6851b3
Merge pull request #2054 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2022-12-29 08:35:51 +01:00
Markus Heiser d531f89316
Merge pull request #2057 from searxng/update_data_update_engine_descriptions.py
Update searx.data - update_engine_descriptions.py
2022-12-29 08:31:19 +01:00
Markus Heiser 76cd808a49
Merge pull request #2053 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2022-12-29 08:28:19 +01:00
dalf 000734cb93 Update searx.data - update_engine_descriptions.py 2022-12-29 01:25:31 +00:00
dalf 36eea5b7cc Update searx.data - update_currencies.py 2022-12-29 01:16:59 +00:00
dalf 66780efda7 Update searx.data - update_wikidata_units.py 2022-12-29 01:16:59 +00:00
dalf de793fb71e Update searx.data - update_ahmia_blacklist.py 2022-12-29 01:16:52 +00:00
Alexandre Flament 0d4896170f
Merge pull request #2047 from dalf/fasttext-predict
Use fasttext-predict instead of fasttext(-wheel)
2022-12-26 20:50:09 +01:00
Alexandre Flament f3515041af Lazy load fasttext-predict 2022-12-26 18:44:23 +00:00
searxng-bot dd125321e4 [translations] update from Weblate
7f3e72fd - 2022-12-20 - Grey <elohim@gmail.com>
5d48e556 - 2022-12-21 - Markus Heiser <markus.heiser@darmarit.de>
56e67eb0 - 2022-12-19 - zDylant <dylantfcs@gmail.com>
28e1abe1 - 2022-12-21 - Markus Heiser <markus.heiser@darmarit.de>
b19d55f7 - 2022-12-17 - YaoSiQian <2229561981@qq.com>
274f7ee9 - 2022-12-17 - Markus Heiser <markus.heiser@darmarit.de>
ab00649b - 2022-12-16 - tents <remendne@pentrens.jp>
2022-12-23 07:14:39 +00:00
Rudis Muiznieks 6804ff048d
Fix: add trailing slash to duckduckgo url
Close #1854
2022-12-22 07:49:58 -06:00
ArtikusHG 735e388cec
Merge branch 'master' into fasttext 2022-12-16 19:43:10 +00:00
ArtikusHG 1f8f8c1e91 Replace langdetect with fasttext 2022-12-16 21:07:39 +02:00
Alexandre Flament 2a51c85672
Merge pull request #1974 from fehho/query-string-deserial
[fix] Deserialize preferences which are explicitly blank strings
2022-12-16 18:56:30 +01:00
searxng-bot 03ce848e04 [translations] update from Weblate
4cf39dae - 2022-12-14 - Markus Heiser <markus.heiser@darmarit.de>
f67457cf - 2022-12-12 - Linerly <linerly@protonmail.com>
dee211c2 - 2022-12-12 - JohnyPeaN <johnypean@gmail.com>
9eeb4290 - 2022-12-12 - Markus Heiser <markus.heiser@darmarit.de>
2022-12-16 07:15:19 +00:00
Alexandre Flament 9e9f57e48b
Merge pull request #1954 from dalf/fix.redis.init.2
[fix] follow up of PR-1856
2022-12-14 07:08:19 +01:00
Markus Heiser ed901ab18e [mod] improve 'Autodetect search language' plugin
- Add documentation to the plugin
- Harmonize FastText language model with SearXNG's language model

Reosurces::

    import fasttext                                    # --> +10 MB
    fasttext.load_model(str(data_dir / 'lid.176.ftz')) # --> +4MB

Suggested-by: @dalf

- To speed up and simplify the deployment use fasttext-wheel instead of fasttext
- Building numpy on the Alpine Linux of docker-images takes ages --> install
  py3-numpy from Alpines package manager (apk)
- Alpine Linux on docker-images (musl libc) do not support fasttext-wheel (gnu
  libc) --> patch Dockerfile and build from fastetxt:

     sed -i s/fasttext-wheel/fasttext/ requirements.txt

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-12-11 11:26:07 +01:00
ArtikusHG 9925a20950 [mod] new plugin: Autodetect search language 2022-12-10 13:11:47 +01:00
Alexandre Flament b971167ced move searx.shared.redisdb to searx.redisdb 2022-12-10 09:26:38 +01:00
Alexandre Flament 34e260f88f [fix] follow up of PR-1856
- the environment variable SEARXNG_REDIS_URL overrides the setting value redis.url
- ./manage sets SEARXNG_REDIS_URL to unix:///usr/local/searxng-redis/run/redis.sock if:
   - the socket exists
   - SEARXNG_REDIS_URL is not already defined

Update of PR #1856

Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2022-12-10 09:26:38 +01:00
searxng-bot 759d53c892 [translations] update from Weblate
1b4600d0 - 2022-12-08 - JohnyPeaN <johnypean@gmail.com>
e3b49827 - 2022-12-04 - Markus Heiser <markus.heiser@darmarit.de>
975217cf - 2022-12-02 - Markus Heiser <markus.heiser@darmarit.de>
2022-12-09 07:15:14 +00:00
Alexandre Flament 269326063a Fix: don't crash when engine or name is missing in settings.yml
SearXNG crashes if the engine or name fields are missing.
With this commit, the app displays an error in the log and keeps loading.

Close #1951
2022-12-04 23:43:59 +01:00
searxng-bot f40d1c0004 [translations] update from Weblate
cc5cf5d8 - 2022-11-30 - Chathura madusanka <mcchathuer216@gmail.com>
97776fbf - 2022-11-29 - Markus Heiser <markus.heiser@darmarit.de>
c3c9f988 - 2022-11-29 - Markus Heiser <markus.heiser@darmarit.de>
a1cf339e - 2022-11-29 - Markus Heiser <markus.heiser@darmarit.de>
4429167c - 2022-11-25 - Markus Heiser <markus.heiser@darmarit.de>
2022-12-02 07:15:43 +00:00
Ryan Kaskel e19218c6a9
Copy client connection kwargs before mutating 2022-11-29 12:15:18 +00:00
Markus Heiser a8359dd4d3
Merge pull request #1981 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2022-11-29 07:48:26 +01:00
Markus Heiser 1b2f1c17f6
Merge pull request #1982 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2022-11-29 07:47:52 +01:00
Markus Heiser 3579a38a2d
Merge pull request #1983 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2022-11-29 07:46:49 +01:00
Markus Heiser 82af2f4460
Merge pull request #1984 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2022-11-29 07:45:51 +01:00
dalf a508cdbd4b Update searx.data - update_engine_descriptions.py 2022-11-29 01:34:42 +00:00
dalf c4512cf64d Update searx.data - update_ahmia_blacklist.py 2022-11-29 01:24:48 +00:00
dalf b0b45fd2d0 Update searx.data - update_currencies.py 2022-11-29 01:24:39 +00:00
dalf a9be73b4da Update searx.data - update_wikidata_units.py 2022-11-29 01:24:38 +00:00
dalf 9fc72419e1 Update searx.data - update_firefox_version.py 2022-11-29 01:24:35 +00:00
searxng-bot 981a2c9626 [translations] update from Weblate
c13727fc - 2022-11-21 - MSI Shafik <shafik.te@gmail.com>
21f638f6 - 2022-11-19 - Markus Heiser <markus.heiser@darmarit.de>
b8ed9e1c - 2022-11-24 - Markus Heiser <markus.heiser@darmarit.de>
2022-11-25 07:16:11 +00:00
fehho 7cab51f98f Deserialize explicitly blank strings
Default behavior of urllib.parse_qs is to discard blank values, causing a preference of none to be deserialized as undefined, using the instance default rather than the selected preference.
2022-11-24 13:14:12 -06:00
searxng-bot 9d06115d07 [translations] update from Weblate
7569f266 - 2022-11-18 - Markus Heiser <markus.heiser@darmarit.de>
6418da9a - 2022-11-18 - mikni <mikni@proton.me>
acc0dc1a - 2022-11-18 - Markus Heiser <markus.heiser@darmarit.de>
6c2beba6 - 2022-11-19 - Alexandre Flament <alex@al-f.net>
b49457e3 - 2022-11-18 - Markus Heiser <markus.heiser@darmarit.de>
2022-11-19 16:21:15 +00:00
searxng-bot 7a39952558 [translations] update from Weblate
f123be21 - 2022-11-13 - Markus Heiser <markus.heiser@darmarit.de>
fa2cfa8f - 2022-11-13 - Markus Heiser <markus.heiser@darmarit.de>
2022-11-18 07:17:51 +00:00
Émilien Devos 46ad32343a Switch back to protobuf for raw HTML 2022-11-11 07:39:48 +00:00
ngosang 78be4b4c70 Fix Google search engine.
- Fix broken links. Resolves #1794
- Fix missing results. Resolves #1829
2022-11-11 07:34:19 +01:00
Ryan Draga 408200c87e [fix] disabling zlibrary due to z-lib.org domain seizure 2022-11-10 21:18:21 +01:00
Markus Heiser ee4475ff4b
Merge pull request #1934 from dalf/fix_user_agent
User agent: don't include the patch number in the Firefox version
2022-11-09 06:50:33 +01:00
Alexandre FLAMENT 73b5a58d9e [FIX] Redis initialization
redis.Redis.from_url(url) doesn't check if the url is valid

Before this commit: actual error are detected later when the client is actually used.
With this commit, client() makes sure to return a valid Redis client or None.

Also, the code makes sure not to log the password of the Redis URL
2022-11-07 22:08:32 +00:00
Alexandre Flament 8f19bdaf17
Merge pull request #1882 from fehho/metacpan
Add MetaCPAN engine
2022-11-07 21:54:11 +01:00
fehho fe351c2802 Add MetaCPAN engine 2022-11-07 08:07:06 -06:00
Evhorizon 1517724615
Update network.py 2022-11-06 20:35:30 +01:00
Alexandre Flament 2dc5c0e1d6
Merge pull request #1928 from searxng/translations_update
Update translations
2022-11-06 11:29:28 +01:00
Alexandre Flament 4fe5463671
Merge pull request #1869 from vgerak/1836-ddg-weather
Fix floating point format in DDG weather humidity
2022-11-05 22:46:50 +01:00
Alexandre FLAMENT d8dee21277 searx.locale: add Dhivehi language 2022-11-05 21:26:15 +00:00
Alexandre FLAMENT 20d951fc5c searx.locales: improve support for languages not supported by babel
* refactor get_translations() to rely on ADDITIONAL_TRANSLATIONS and LOCALE_BEST_MATCH
* update RTL_LOCALES for languages in ADDITIONAL_TRANSLATIONS
2022-11-05 21:26:07 +00:00
Alexandre Flament e473addaff User agent: don't include the patch number in the Firefox version
The Firefox version in the user agent doesn't include the patch version: 106.0 not 106.0.2

Close #1914
2022-11-05 22:04:37 +01:00
Alexandre FLAMENT e92755d358 Initialize Redis in searx/webapp.py
settings.yml:
* The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0
* The default URL is now "false"

The default URL makes the log difficult to deal with:
if the admin didn't install a Redis instance, the logs record a false error.

It worked before because SearXNG initialized the Redis connection when the limiter started.

In this commit, SearXNG initializes Redis in searx/webapp.py
so various components can use Redis without taking care of the initialization step.
2022-11-05 17:45:52 +01:00
Alexandre Flament fe419e355b The checker requires Redis
Remove the abstraction in searx.shared.SharedDict.
Implement a basic and dedicated scheduler for the checker using a Redis script.
2022-11-05 12:04:50 +01:00
searxng-bot 7438e48d13 [translations] update from Weblate
41ebf8e7 - 2022-10-31 - Landhoo School Students <landhooschoolstudents@gmail.com>
1de65e6a - 2022-10-29 - Markus Heiser <markus.heiser@darmarit.de>
fff11c22 - 2022-10-30 - Markus Heiser <markus.heiser@darmarit.de>
f677049f - 2022-10-30 - Markus Heiser <markus.heiser@darmarit.de>
ebf69b4a - 2022-10-30 - Markus Heiser <markus.heiser@darmarit.de>
a21b6221 - 2022-11-02 - Markus Heiser <markus.heiser@darmarit.de>
ac384800 - 2022-11-03 - ptomljanovic <phill2605@gmail.com>
c4736fd4 - 2022-11-03 - alexfs2015 <alex04fs@gmail.com>
1b967129 - 2022-10-31 - Landhoo School Students <landhooschoolstudents@gmail.com>
2022-11-04 07:18:35 +00:00
Markus Heiser fc9986de0a
Merge pull request #1905 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2022-10-29 08:22:43 +02:00
Markus Heiser 3f1d594c23
Merge pull request #1906 from searxng/update_data_update_external_bangs.py
Update searx.data - update_external_bangs.py
2022-10-29 08:21:33 +02:00
Markus Heiser a9deead102
Merge pull request #1907 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2022-10-29 08:20:53 +02:00
Markus Heiser d49ccb54ea
Merge pull request #1908 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2022-10-29 08:20:27 +02:00
Markus Heiser fa59ff9b78
Merge pull request #1909 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2022-10-29 08:19:56 +02:00
dalf e70589c598 Update searx.data - update_engine_descriptions.py 2022-10-29 01:56:57 +00:00
dalf 985a2a87a6 Update searx.data - update_currencies.py 2022-10-29 01:46:40 +00:00
dalf c0ecc2f865 Update searx.data - update_wikidata_units.py 2022-10-29 01:46:38 +00:00
dalf c24abc2f8c Update searx.data - update_firefox_version.py 2022-10-29 01:46:34 +00:00
dalf f0c7898445 Update searx.data - update_external_bangs.py 2022-10-29 01:46:27 +00:00
dalf 799c5270ae Update searx.data - update_ahmia_blacklist.py 2022-10-29 01:46:24 +00:00
Markus Heiser 5a181ea1f5
Merge pull request #1901 from searxng/dependabot/npm_and_yarn/searx/static/themes/simple/master/svgo-3.0.0
Bump svgo from 2.8.0 to 3.0.0 in /searx/static/themes/simple
2022-10-28 10:22:56 +02:00
Markus Heiser d913f0d8aa [build] /static 2022-10-28 10:17:56 +02:00
searxng-bot bc3db6fd16 [translations] update from Weblate
bf52cdbc - 2022-10-22 - Markus Heiser <markus.heiser@darmarit.de>
fbb36c76 - 2022-10-22 - forvirretfrukt <lillelilli87@gmail.com>
18cc8a36 - 2022-10-22 - Markus Heiser <markus.heiser@darmarit.de>
7314d2b6 - 2022-10-24 - Markus Heiser <markus.heiser@darmarit.de>
b25147b9 - 2022-10-27 - Markus Heiser <markus.heiser@darmarit.de>
d39ade65 - 2022-10-24 - Markus Heiser <markus.heiser@darmarit.de>
cb835cdc - 2022-10-24 - Markus Heiser <markus.heiser@darmarit.de>
563f4e0b - 2022-10-26 - Markus Heiser <markus.heiser@darmarit.de>
a698a2e9 - 2022-10-24 - Markus Heiser <markus.heiser@darmarit.de>
df5ce281 - 2022-10-22 - Markus Heiser <markus.heiser@darmarit.de>
55b369a3 - 2022-10-22 - Markus Heiser <markus.heiser@darmarit.de>
3eddf243 - 2022-10-24 - zaioti <zaioti@tuta.io>
edd3871d - 2022-10-24 - zaioti <zaioti@tuta.io>
5aab9f7b - 2022-10-22 - Markus Heiser <markus.heiser@darmarit.de>
0b2cfe68 - 2022-10-22 - tents <remendne@pentrens.jp>
732b7865 - 2022-10-22 - Markus Heiser <markus.heiser@darmarit.de>
d652c20f - 2022-10-22 - Markus Heiser <markus.heiser@darmarit.de>
0bcaaa8d - 2022-10-24 - Markus Heiser <markus.heiser@darmarit.de>
6fd87821 - 2022-10-26 - Markus Heiser <markus.heiser@darmarit.de>
618f213c - 2022-10-21 - Masterix <masterix@duck.com>
30d38ee9 - 2022-10-24 - Markus Heiser <markus.heiser@darmarit.de>
a934ce3b - 2022-10-23 - Markus Heiser <markus.heiser@darmarit.de>
b696c22e - 2022-10-23 - Markus Heiser <markus.heiser@darmarit.de>
6ee1efe2 - 2022-10-23 - Jeewaka Fernando <Jeewakaphotography@gmail.com>
2022-10-28 07:18:50 +00:00
dependabot[bot] 3cade9a7eb
Bump svgo from 2.8.0 to 3.0.0 in /searx/static/themes/simple
Bumps [svgo](https://github.com/svg/svgo) from 2.8.0 to 3.0.0.
- [Release notes](https://github.com/svg/svgo/releases)
- [Changelog](https://github.com/svg/svgo/blob/main/CHANGELOG-old.md)
- [Commits](https://github.com/svg/svgo/compare/v2.8.0...v3.0.0)

---
updated-dependencies:
- dependency-name: svgo
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-10-28 07:13:58 +00:00
searxng-bot 18ba8628dd [translations] update from Weblate
c7fddee3 - 2022-10-15 - Linerly <linerly@protonmail.com>
f559a626 - 2022-10-17 - Markus Heiser <markus.heiser@darmarit.de>
fe7af662 - 2022-10-17 - Mateusz Bączek <mateusz.baczek1998@gmail.com>
8a7c3da9 - 2022-10-16 - Peter Martin <weblate@pe7er.com>
a84dad5c - 2022-10-16 - Franco Longo <longofrancoale@gmail.com>
9ee1ac8b - 2022-10-20 - prljav <marin.ruskica@gmail.com>
b91bedc4 - 2022-10-16 - Peter Martin <weblate@pe7er.com>
dff99ef4 - 2022-10-16 - Peter Martin <weblate@pe7er.com>
cd639f6c - 2022-10-17 - Markus Heiser <markus.heiser@darmarit.de>
49507a60 - 2022-10-14 - Markus Heiser <markus.heiser@darmarit.de>
ad305582 - 2022-10-16 - Peter Martin <weblate@pe7er.com>
19e97075 - 2022-10-17 - Markus Heiser <markus.heiser@darmarit.de>
97581f70 - 2022-10-14 - Markus Heiser <markus.heiser@darmarit.de>
fcd52cea - 2022-10-14 - Alexandre Flament <alex@al-f.net>
2022-10-21 07:32:09 +00:00
Vasilis Gerakaris 947b62c9d5
Fix floating point format in DDG weather humidity
Fixes #1836
2022-10-20 11:44:17 +03:00
Alexandre FLAMENT 035bc507ec [fix] startpage engine 2022-10-14 18:27:53 +00:00
Alexandre Flament 32e8c2cf09 searx.network: add "verify" option to the networks
Each network can define a verify option:
* false to disable certificate verification
* a path to existing certificate.

SearXNG uses SSL_CERT_FILE and SSL_CERT_DIR when they are defined
see https://www.python-httpx.org/environment_variables/#ssl_cert_file
2022-10-14 13:59:22 +00:00
searxng-bot 6e86f33342 [translations] update from Weblate
eee3d5e2 - 2022-10-12 - Markus Heiser <markus.heiser@darmarit.de>
c4eb9818 - 2022-10-11 - Markus Heiser <markus.heiser@darmarit.de>
27423f5b - 2022-10-12 - Markus Heiser <markus.heiser@darmarit.de>
9e8b7f86 - 2022-10-11 - No4vick <MineBor1@yandex.ru>
1203aa6a - 2022-10-11 - KEINOS <github@keinos.com>
22c78af5 - 2022-10-12 - Markus Heiser <markus.heiser@darmarit.de>
6a4776f8 - 2022-10-11 - Markus Heiser <markus.heiser@darmarit.de>
c00bdc7d - 2022-10-11 - Markus Heiser <markus.heiser@darmarit.de>
e5a39537 - 2022-10-09 - Kayra Uylar <k.uylar@outlook.com>
e3fad460 - 2022-10-11 - Markus Heiser <markus.heiser@darmarit.de>
da5f0bc3 - 2022-10-11 - Surepusofu Arutemu <crexlight@gmail.com>
5704a98b - 2022-10-11 - Markus Heiser <markus.heiser@darmarit.de>
f3ef3ebb - 2022-10-11 - Markus Heiser <markus.heiser@darmarit.de>
620636c1 - 2022-10-11 - Markus Heiser <markus.heiser@darmarit.de>
767b3bb2 - 2022-10-08 - KEINOS <github@keinos.com>
60d38c87 - 2022-10-11 - Markus Heiser <markus.heiser@darmarit.de>
2022-10-14 07:38:20 +00:00
Markus Heiser 23d797ee6f Update searx.data - update_osm_keys_tags.py
Patch was generated by::

  ./manage pyenv.cmd searxng_extra/update/update_osm_keys_tags.py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-10-11 11:49:06 +02:00
searxng-bot 325eadfd04 [translations] update from Weblate
70909b19 - 2022-10-04 - Yan Lyra <yanlyra3@gmail.com>
acf1f0aa - 2022-10-03 - George Kashkosh <kash.george@gmail.com>
94ce447f - 2022-10-01 - Zaid Aiman <zaidaiman@gmail.com>
59959050 - 2022-09-30 - tjvfxtiwdr <tjvfxtiwdr@daimashili.com>
e27a5c79 - 2022-09-30 - a <a@daimashili.com>
2c05470f - 2022-09-30 - tjvfxtiwdr <tjvfxtiwdr@daimashili.com>
53644516 - 2022-10-01 - random <thrizem+wnd43@gmail.com>
2022-10-07 07:30:33 +00:00