* Docker: add UWSGI_WORKERS and UWSGI_THREAD.
UWSGI_WORKERS specifies the number of process.
UWSGI_THREADS specifies the number of threads.
The Docker convention is to specify the whole configuration
through environment variables. While not done in SearXNG, these two
additional variables allows admins to skip uwsgi.ini
In additional, https://github.com/searxng/preview-environments starts Docker
without additional files through searxng-helm-chat.
Each instance consumes 1Go of RAM which is a lot especially when there are a
lot of instances / pull requests.
* [scripts] add environments UWSGI_WORKERS and UWSGI_THREADS
- UWSGI_WORKERS specifies the number of process.
- UWSGI_THREADS specifies the number of threads.
Templates for uwsgi scripts can be tested by::
UWSGI_WORKERS=8 UWSGI_THREADS=9 \
./utils/searxng.sh --cmd\
eval "echo \"$(cat utils/templates/etc/uwsgi/*/searxng.ini*)\""\
| grep "workers\|threads"
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---------
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
- the option server:public_instance lacks some documentation
- the processing of this option belongs in the limiter and not
in botdetection module
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch was inspired by the discussion around PR-2882 [2]. The goals of this
patch are:
1. Convert plugin searx.plugin.limiter to normal code [1]
2. isolation of botdetection from the limiter [2]
3. searx/{tools => botdetection}/config.py and drop searx.tools
4. in URL /config, 'limiter.enabled' is true only if the limiter is really
enabled (Redis is available).
This patch moves all the code that belongs to botdetection into namespace
searx.botdetection and code that belongs to limiter is placed in namespace
searx.limiter.
Tthe limiter used to be a plugin at some point botdetection was added, it was
not a plugin. The modularization of these two components was long overdue.
With the clear modularization, the documentation could then also be organized
according to the architecture.
[1] https://github.com/searxng/searxng/pull/2882
[2] https://github.com/searxng/searxng/pull/2882#issuecomment-1741716891
To test:
- check the app works without the limiter, check `/config`
- check the app works with the limiter and with the token, check `/config`
- make docs.live .. and read
- http://0.0.0.0:8000/admin/searx.limiter.html
- http://0.0.0.0:8000/src/searx.botdetection.html#botdetection
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To test this PR run a local instance and try to query page 51:
http://127.0.0.1:8888/search?q=foo&pageno=51
A parameter exception will be raised:
searx.exceptions.SearxParameterException: Invalid value "51" for parameter pageno
And the client will receive a HTTP 400 (Bad request).
Closes https://github.com/searxng/searxng/issues/2972
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In python versions <py3.10 there is an issue with an undocumented method
HTMLParser.error() [1][2] that was deprecated in Python 3.4 and removed
in Python 3.5.
To be compatible to higher versions (>=py3.10) an error method is implemented
which throws an AssertionError exception like the higher Python versions do [3].
[1] https://github.com/python/cpython/issues/76025
[2] https://bugs.python.org/issue31844
[3] https://github.com/python/cpython/pull/8562
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch implements a table of ranges. A *range* is a series of integers,
e.g. 0-10 with start value 0 and end value 10.
Start and end values are stored in a two-column table, the RangeTable. This
table is stored in Redis in a sorted list and a ZRANGEBYSCORE query can be used
to determine if a particular integer is contained in one of the ranges in the
table.
The ZRANGEBYSCORE query scales with O(log(N)). To make it more clear: a query in
a table with 1 million ranges takes just twice as long as the query in a table
with 1000 ranges.
A RangeTable is excellent for IP ranges and ultra-fast queries to see if an
IP (which is nothing but an integer) is contained in one of the ranges of the
table.
Typical applications are e.g. large IP block lists or network ranges.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
When ``searx/static/themes/simple/node_modules`` are installed the test from
black runs into this path / Error::
$ make test
TEST [yamllint] $YAMLLINT_FILES
TEST [black] $BLACK_TARGETS
--- searx/static/themes/simple/node_modules/flatted/python/test.py 2023-10-13 10:09:50.460026 +0000
+++ searx/static/themes/simple/node_modules/flatted/python/test.py 2023-10-13 10:10:04.974335 +0000
@@ -1,9 +1,11 @@
from flatted import stringify as _stringify, parse
...
black --exclude option is not a comma separated list, its a regular expression /
see:
$ black --help
...
--exclude TEXT A regular expression that matches files and
directories that should be excluded on recursive
searches ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
DDG's bot detection is sensitive to the vqd value. For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.
If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.
Requests from IPs in this block list run into timeouts.
Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).
Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.
Related:
- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Some search terms do not have results and therefore no vqd value
BTW: remove a leftover from 9197efa
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].
From reverse engineering:
- The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite
does not like this referer anymore!
- The 'Referer' header is only set on second and follow up pages but not on the
first page
- The vqd value is not needed on the first page, the ddg-lite client sets this
value only on follow up pages / this can help to reduce the vqd requests from
SearXNG.
Related to 'Referer' header & ddg requests:
[1] https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081
Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The change in the hotkey mechanism introduced in 317db5b04 does not allow
configuration via `settings.yml`. This commit adds that functionality.
Closes: #2898
Markus Heiser
searxng-bot
dependabot[bot]
Alexandre Flament
Paul Braeuning
Jinyuan Huang
sev
dalf
Bnyro
Émilien (perso)
Hackurei
Umut Ucak
Alex Balgavy
Hackurei