Commit Graph

7741 Commits

Author SHA1 Message Date
Jinyuan Huang 727de1c4ef [build] /static 2023-11-10 13:28:28 +00:00
Jinyuan Huang 8adc7b0433 add a "copy" button to the "search-url" sidebar 2023-11-10 13:28:04 +00:00
searxng-bot 96ccec1aa6 [translations] update from Weblate
acdcdace8 - 2023-11-01 - Ivan Gabaldon <admin@inetol.net>
31a1602a9 - 2023-10-29 - return42 <markus.heiser@darmarit.de>
a1d1070d3 - 2023-10-29 - return42 <markus.heiser@darmarit.de>
2023-11-03 10:01:28 +01:00
dependabot[bot] dd0ccd7904 Bump selenium from 4.14.0 to 4.15.1
Bumps [selenium](https://github.com/SeleniumHQ/Selenium) from 4.14.0 to 4.15.1.
- [Release notes](https://github.com/SeleniumHQ/Selenium/releases)
- [Commits](https://github.com/SeleniumHQ/Selenium/commits)

---
updated-dependencies:
- dependency-name: selenium
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-03 10:00:54 +01:00
dependabot[bot] baef837eb6 Bump linuxdoc from 20230827 to 20231020
Bumps [linuxdoc](https://github.com/return42/linuxdoc) from 20230827 to 20231020.
- [Changelog](https://github.com/return42/linuxdoc/blob/master/CHANGELOG)
- [Commits](https://github.com/return42/linuxdoc/commits)

---
updated-dependencies:
- dependency-name: linuxdoc
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-03 09:59:06 +01:00
Markus Heiser b5a8ddfec8
Revert "[fix] limit maximum page number of a search query to page 50." (#2979)
This reverts commit 7e2e335dd1.
2023-11-01 16:55:20 +00:00
Markus Heiser d13a8f6453 [mod] document server:public_instance & remove it out of the botdetection
- the option server:public_instance lacks some documentation
- the processing of this option belongs in the limiter and not
  in botdetection module

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01 06:44:56 +01:00
Markus Heiser fd814aac86 [mod] isolation of botdetection from the limiter
This patch was inspired by the discussion around PR-2882 [2].  The goals of this
patch are:

1. Convert plugin searx.plugin.limiter to normal code [1]
2. isolation of botdetection from the limiter [2]
3. searx/{tools => botdetection}/config.py and drop searx.tools
4. in URL /config, 'limiter.enabled' is true only if the limiter is really
   enabled (Redis is available).

This patch moves all the code that belongs to botdetection into namespace
searx.botdetection and code that belongs to limiter is placed in namespace
searx.limiter.

Tthe limiter used to be a plugin at some point botdetection was added, it was
not a plugin.  The modularization of these two components was long overdue.
With the clear modularization, the documentation could then also be organized
according to the architecture.

[1] https://github.com/searxng/searxng/pull/2882
[2] https://github.com/searxng/searxng/pull/2882#issuecomment-1741716891

To test:

- check the app works without the limiter, check `/config`
- check the app works with the limiter and with the token, check `/config`
- make docs.live .. and read
  - http://0.0.0.0:8000/admin/searx.limiter.html
  - http://0.0.0.0:8000/src/searx.botdetection.html#botdetection

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-01 06:44:56 +01:00
sev b05a15540e Check `public_instance` in simple theme
Fix #2975
2023-10-31 10:40:19 +01:00
Markus Heiser 7e2e335dd1 [fix] limit maximum page number of a search query to page 50.
To test this PR run a local instance and try to query page 51:

    http://127.0.0.1:8888/search?q=foo&pageno=51

A parameter exception will be raised:

    searx.exceptions.SearxParameterException: Invalid value "51" for parameter pageno

And the client will receive a HTTP 400 (Bad request).

Closes https://github.com/searxng/searxng/issues/2972

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-29 16:17:25 +01:00
Alexandre Flament 9aeae2142b Add support for Python 3.12 2023-10-29 11:25:58 +01:00
dalf b63841f0a1 Update searx.data - update_engine_traits.py 2023-10-29 09:04:24 +01:00
dependabot[bot] 12a83b8cb3 Bump babel from 2.13.0 to 2.13.1
Bumps [babel](https://github.com/python-babel/babel) from 2.13.0 to 2.13.1.
- [Release notes](https://github.com/python-babel/babel/releases)
- [Changelog](https://github.com/python-babel/babel/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-babel/babel/compare/v2.13.0...v2.13.1)

---
updated-dependencies:
- dependency-name: babel
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-29 09:03:43 +01:00
dependabot[bot] 15c8301e52 Bump sphinx-tabs from 3.4.1 to 3.4.4
Bumps [sphinx-tabs](https://github.com/executablebooks/sphinx-tabs) from 3.4.1 to 3.4.4.
- [Release notes](https://github.com/executablebooks/sphinx-tabs/releases)
- [Changelog](https://github.com/executablebooks/sphinx-tabs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/executablebooks/sphinx-tabs/compare/v3.4.1...v3.4.4)

---
updated-dependencies:
- dependency-name: sphinx-tabs
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-29 08:58:07 +01:00
dependabot[bot] cf605cf899 Bump uvloop from 0.18.0 to 0.19.0
Bumps [uvloop](https://github.com/MagicStack/uvloop) from 0.18.0 to 0.19.0.
- [Release notes](https://github.com/MagicStack/uvloop/releases)
- [Commits](https://github.com/MagicStack/uvloop/compare/v0.18.0...v0.19.0)

---
updated-dependencies:
- dependency-name: uvloop
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-29 08:54:32 +01:00
dalf a9d778ee0f Update searx.data - update_wikidata_units.py 2023-10-29 08:45:15 +01:00
dalf 92efb941d8 Update searx.data - update_firefox_version.py 2023-10-29 08:44:59 +01:00
dalf fa494b9be3 Update searx.data - update_currencies.py 2023-10-29 08:44:35 +01:00
dalf 1b92ea4e70 Update searx.data - update_engine_descriptions.py 2023-10-29 08:44:17 +01:00
dalf 55e7cb3e33 Update searx.data - update_ahmia_blacklist.py 2023-10-29 08:17:29 +01:00
dependabot[bot] dbe94f111a Bump pylint from 3.0.1 to 3.0.2
Bumps [pylint](https://github.com/pylint-dev/pylint) from 3.0.1 to 3.0.2.
- [Release notes](https://github.com/pylint-dev/pylint/releases)
- [Commits](https://github.com/pylint-dev/pylint/compare/v3.0.1...v3.0.2)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-27 10:47:10 +02:00
searxng-bot 6af8644ead [translations] update from Weblate
4e5e5db44 - 2023-10-26 - quenty_occitania <quentinantonin@free.fr>
e1a8d3508 - 2023-10-26 - quenty_occitania <quentinantonin@free.fr>
84bddfb89 - 2023-10-26 - return42 <markus.heiser@darmarit.de>
d67a4114d - 2023-10-26 - quenty_occitania <quentinantonin@free.fr>
62fe8e328 - 2023-10-26 - return42 <markus.heiser@darmarit.de>
6e37ab975 - 2023-10-26 - quenty_occitania <quentinantonin@free.fr>
2cdab3247 - 2023-10-25 - SomeTr <SomeTr@users.noreply.translate.codeberg.org>
cf7ea7234 - 2023-10-23 - clsty <celestial.y@outlook.com>
0ea313893 - 2023-10-23 - return42 <markus.heiser@darmarit.de>
22151e440 - 2023-10-23 - return42 <markus.heiser@darmarit.de>
e4eaf42b6 - 2023-10-22 - clsty <celestial.y@outlook.com>
2023-10-27 09:23:23 +02:00
dependabot[bot] 526d5c7b30 Bump uvloop from 0.17.0 to 0.18.0
Bumps [uvloop](https://github.com/MagicStack/uvloop) from 0.17.0 to 0.18.0.
- [Release notes](https://github.com/MagicStack/uvloop/releases)
- [Commits](https://github.com/MagicStack/uvloop/compare/v0.17.0...v0.18.0)

---
updated-dependencies:
- dependency-name: uvloop
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-22 11:01:41 +02:00
Markus Heiser ef56e1d684 [fix] HTMLParser: undocumented not implemented method
In python versions <py3.10 there is an issue with an undocumented method
HTMLParser.error() [1][2] that was deprecated in Python 3.4 and removed
in Python 3.5.

To be compatible to higher versions (>=py3.10) an error method is implemented
which throws an AssertionError exception like the higher Python versions do [3].

[1] https://github.com/python/cpython/issues/76025
[2] https://bugs.python.org/issue31844
[3] https://github.com/python/cpython/pull/8562

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-22 10:35:02 +02:00
searxng-bot 01b5b9cb8e [translations] update from Weblate
2325f1583 - 2023-10-20 - return42 <markus.heiser@darmarit.de>
5090c6a8e - 2023-10-18 - return42 <markus.heiser@darmarit.de>
3a38219d8 - 2023-10-18 - return42 <markus.heiser@darmarit.de>
94a9f4164 - 2023-10-16 - return42 <markus.heiser@darmarit.de>
bdbeb4b30 - 2023-10-16 - return42 <markus.heiser@darmarit.de>
f9b483f48 - 2023-10-16 - return42 <markus.heiser@darmarit.de>
7f1ca1997 - 2023-10-16 - return42 <markus.heiser@darmarit.de>
c5a701dc4 - 2023-10-14 - alexgabi <alexgabi@disroot.org>
2023-10-20 10:44:11 +02:00
Markus Heiser b013cbb4cc [build] /static 2023-10-13 12:56:29 +02:00
Bnyro df2e41c150 [fix] search.js: crash on homepage when setting form listeners 2023-10-13 12:56:29 +02:00
Markus Heiser 70521339d9 [fix] black --exclude option is a regular expression
When ``searx/static/themes/simple/node_modules`` are installed the test from
black runs into this path / Error::

    $ make test
    TEST      [yamllint] $YAMLLINT_FILES
    TEST      [black] $BLACK_TARGETS
    --- searx/static/themes/simple/node_modules/flatted/python/test.py	2023-10-13 10:09:50.460026 +0000
    +++ searx/static/themes/simple/node_modules/flatted/python/test.py	2023-10-13 10:10:04.974335 +0000
    @@ -1,9 +1,11 @@
    from flatted import stringify as _stringify, parse
    ...

black --exclude option is not a comma separated list, its a regular expression /
see:

    $ black --help
    ...
    --exclude TEXT   A regular expression that matches files and
                     directories that should be excluded on recursive
                     searches ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-13 12:44:18 +02:00
dependabot[bot] 7ee1d035bf Bump flask from 2.3.3 to 3.0.0
Bumps [flask](https://github.com/pallets/flask) from 2.3.3 to 3.0.0.
- [Release notes](https://github.com/pallets/flask/releases)
- [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/flask/compare/2.3.3...3.0.0)

---
updated-dependencies:
- dependency-name: flask
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-13 11:49:17 +02:00
Émilien (perso) 4280318fc5 fixing results parsing brave 2023-10-13 11:47:30 +02:00
dependabot[bot] 7f59b6d62e Bump flask-babel from 3.1.0 to 4.0.0
Bumps [flask-babel](https://github.com/python-babel/flask-babel) from 3.1.0 to 4.0.0.
- [Release notes](https://github.com/python-babel/flask-babel/releases)
- [Changelog](https://github.com/python-babel/flask-babel/blob/master/CHANGELOG)
- [Commits](https://github.com/python-babel/flask-babel/compare/v3.1.0...v4.0.0)

---
updated-dependencies:
- dependency-name: flask-babel
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-13 09:51:30 +02:00
dependabot[bot] 32d52cd856 Bump selenium from 4.13.0 to 4.14.0
Bumps [selenium](https://github.com/SeleniumHQ/Selenium) from 4.13.0 to 4.14.0.
- [Release notes](https://github.com/SeleniumHQ/Selenium/releases)
- [Commits](https://github.com/SeleniumHQ/Selenium/compare/selenium-4.13.0...selenium-4.14.0)

---
updated-dependencies:
- dependency-name: selenium
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-13 09:33:04 +02:00
searxng-bot ac82daf58d [translations] update from Weblate
74e401e68 - 2023-10-09 - return42 <markus.heiser@darmarit.de>
897dd8db1 - 2023-10-09 - return42 <markus.heiser@darmarit.de>
6ed046a90 - 2023-10-09 - tentsbet <remendne@pentrens.jp>
815ecb336 - 2023-10-09 - return42 <markus.heiser@darmarit.de>
65d9a0c2f - 2023-10-07 - return42 <markus.heiser@darmarit.de>
3ec249ef9 - 2023-10-07 - return42 <markus.heiser@darmarit.de>
53dc6c108 - 2023-10-07 - return42 <markus.heiser@darmarit.de>
2023-10-13 09:20:54 +02:00
Hackurei efada7cba2 [fix] hackernews keyerror problem 2023-10-13 08:16:47 +02:00
Hackurei af071121de [fix] imgur - incorrect wikidata id 2023-10-12 09:14:00 +02:00
Markus Heiser 14323d683f [fix] ddg-lite & ddg-extra: don't send empty vqd value
DDG's bot detection is sensitive to the vqd value.  For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.

If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).

Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.

Related:

- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-12 08:52:28 +02:00
Umut Ucak 24561370b7 readme typo 2023-10-11 06:19:31 +02:00
Markus Heiser 3388441917 [fix] ddg-lite vqd value: some search terms do not have a vqd value
Some search terms do not have results and therefore no vqd value

BTW: remove a leftover from 9197efa

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-10 09:12:30 +02:00
Markus Heiser 9197efa2a7 [fix] duckduckgo lite engine: set HTTP header 'Referer'
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].

From reverse engineering:

- The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite
  does not like this referer anymore!

- The 'Referer' header is only set on second and follow up pages but not on the
  first page

- The vqd value is not needed on the first page, the ddg-lite client sets this
  value only on follow up pages / this can help to reduce the vqd requests from
  SearXNG.

Related to 'Referer' header & ddg requests:

[1] https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081

Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-10 08:40:53 +02:00
Bnyro fa5b2a7948 [mod] yacy: use official instance by default and fix crashes 2023-10-09 20:50:24 +02:00
Alex Balgavy 5d53aac20f [mod] add hotkeys option to settings.yml
The change in the hotkey mechanism introduced in 317db5b04 does not allow
configuration via `settings.yml`.  This commit adds that functionality.

Closes: #2898
2023-10-09 18:13:00 +02:00
Hackurei ff78b1a902 [feat] implement hackernews engine - news.ycombinator.com 2023-10-09 14:00:04 +02:00
Aine 213cb74378 [fix] matrixrooms add proper MRS integration
Related:

- https://github.com/searxng/searxng/issues/2918
2023-10-09 13:25:13 +02:00
Bnyro 48cb58bd2e [feat] duckduckgo: support for videos and news 2023-10-09 06:53:43 +02:00
Bnyro c3ab49cd90 [fix] kickass: crash when no results 2023-10-07 11:48:23 +02:00
Bnyro f22daf8b47 [mod] piped: always show video length if available 2023-10-07 11:45:46 +02:00
Bnyro ce270961e8 [feat] engine: implementation of mastodon 2023-10-06 10:58:23 +02:00
dependabot[bot] e5e3e22f7c Bump nose2[coverage_plugin] from 0.13.0 to 0.14.0
Bumps [nose2[coverage_plugin]](https://github.com/nose-devs/nose2) from 0.13.0 to 0.14.0.
- [Changelog](https://github.com/nose-devs/nose2/blob/main/docs/changelog.rst)
- [Commits](https://github.com/nose-devs/nose2/compare/0.13.0...0.14.0)

---
updated-dependencies:
- dependency-name: nose2[coverage_plugin]
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-06 10:29:40 +02:00
dependabot[bot] 60fecf5125 Bump setproctitle from 1.3.2 to 1.3.3
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.3.2 to 1.3.3.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/compare/version-1.3.2...version-1.3.3)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-06 10:29:12 +02:00
dependabot[bot] f7ca979c8a Bump pylint from 2.17.6 to 3.0.1
Bumps [pylint](https://github.com/pylint-dev/pylint) from 2.17.6 to 3.0.1.
- [Release notes](https://github.com/pylint-dev/pylint/releases)
- [Commits](https://github.com/pylint-dev/pylint/compare/v2.17.6...v3.0.1)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-06 10:22:57 +02:00