jandieman
/
searxng
mirror of https://github.com/searxng/searxng.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
searxng/docs/src/searx.botdetection.rst

49 lines
797 B
ReStructuredText
Raw Normal View History

[mod] limiter -> botdetection: modularization and documentation In order to be able to meet the outstanding requirements, the implementation is modularized and supplemented with documentation. This patch does not contain functional change, except it fixes issue #2455 ---- Aktivate limiter in the settings.yml and simulate a bot request by:: curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \ -H 'Accept: text/html' -H 'User-Agent: xyz' \ -H 'Accept-Encoding: gzip' \ 'http://127.0.0.1:8888/search?q=foo' In the LOG: DEBUG searx.botdetection.link_token : missing ping for this request: ..... Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time before you get a "Too Many Requests" response. Closes: https://github.com/searxng/searxng/issues/2455 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-23 16:16:37 +00:00
.. _botdetection:
=============
Bot Detection
=============
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.botdetection
:members:
.. automodule:: searx.botdetection.limiter
:members:
[mod] limiter: blocklist and passlist (ip_lists) A blocklist and a passlist can be configured in /etc/searxng/limiter.toml:: [botdetection.ip_lists] pass_ip = [ '51.15.252.168', # IPv4 of check.searx.space ] block_ip = [ '93.184.216.34', # IPv4 of example.org ] Closes: https://github.com/searxng/searxng/issues/2127 Closes: https://github.com/searxng/searxng/pull/2129 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-03 11:43:34 +00:00
.. automodule:: searx.botdetection.ip_lists
:members:
[mod] limiter -> botdetection: modularization and documentation In order to be able to meet the outstanding requirements, the implementation is modularized and supplemented with documentation. This patch does not contain functional change, except it fixes issue #2455 ---- Aktivate limiter in the settings.yml and simulate a bot request by:: curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \ -H 'Accept: text/html' -H 'User-Agent: xyz' \ -H 'Accept-Encoding: gzip' \ 'http://127.0.0.1:8888/search?q=foo' In the LOG: DEBUG searx.botdetection.link_token : missing ping for this request: ..... Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time before you get a "Too Many Requests" response. Closes: https://github.com/searxng/searxng/issues/2455 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-23 16:16:37 +00:00
Rate limit
==========
.. automodule:: searx.botdetection.ip_limit
:members:
.. automodule:: searx.botdetection.link_token
:members:
Probe HTTP headers
==================
.. automodule:: searx.botdetection.http_accept
:members:
.. automodule:: searx.botdetection.http_accept_encoding
:members:
.. automodule:: searx.botdetection.http_accept_language
:members:
.. automodule:: searx.botdetection.http_connection
:members:
.. automodule:: searx.botdetection.http_user_agent
:members: