[POC] limiter: change PING of link_token mehtod from CSS to <img>

while PR #2357 [1] was being implemented the question came up:

    would be better to change the PING resource from CSS to an image so that
    some terminal based browser may still able to pass the test [1]

This patch implements a POC in where a <img src=token> tag is loaded instaed a
CSS.

To test this patch activate limiter and link_token method [3] and start a
developer instance::

    make run

In your terminal browser open http://127.0.0.1:8888/search?q=foo

If the browser is suitable for the link_token method, it loads the image and the
following messages appear::

    DEBUG   searx.botdetection.limiter    : OK 127.0.0.1/32: /clientft61aak7fzyu6o6v.svg ...
    DEBUG   searx.botdetection.link_token : token is valid --> True
    DEBUG   searx.botdetection.link_token : store ping_key for (client) network 127.0.0.1/32 (IP 127.0.0.1) -> SearXNG_limiter.ping[...]

Browsers that do not load images will be blocked: If you try by example::

    lynx http://127.0.0.1:8888/search?q=foo

you will see a WARNING message like::

    WARNING searx.botdetection.link_token : missing ping (IP: 127.0.0.1/32) / request: SearXNG_limiter.ping[...]

----

[1] 80aaef6c95
[2] https://github.com/searxng/searxng/pull/2357#issuecomment-1574898834
[3] activate limiter and link_token method

```diff
diff --git a/searx/botdetection/limiter.toml b/searx/botdetection/limiter.toml
index 71a231e8f..7e1dba755 100644
--- a/searx/botdetection/limiter.toml
+++ b/searx/botdetection/limiter.toml
@@ -17,6 +17,6 @@ ipv6_prefix = 48
 filter_link_local = false

 # acrivate link_token method in the ip_limit method
-link_token = false
+link_token = true

diff --git a/searx/settings.yml b/searx/settings.yml
index a82a3432d..e7b983afc 100644
--- a/searx/settings.yml
+++ b/searx/settings.yml
@@ -73,7 +73,7 @@ server:
   # public URL of the instance, to ensure correct inbound links. Is overwritten
   # by ${SEARXNG_URL}.
   base_url: false  # "http://example.com/location"
-  limiter: false  # rate limit the number of request on the instance, block some bots
+  limiter: true  # rate limit the number of request on the instance, block some bots

   # If your instance owns a /etc/searxng/settings.yml file, then set the following
   # values there.
```

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

.editorconfig

@@ -16,9 +16,6 @@ max_line_length = 119
 [*.html]
 indent_size = 4
 
-[*.js]
-indent_size = 2
-
 [*.json]
 indent_size = 4
 insert_final_newline = ignore

.github/ISSUE_TEMPLATE/engine-request.md

@@ -20,7 +20,7 @@ assignees: ''
 **How can SearXNG fetch the information from this engine?**
 <!-- List API URL, example code (using the correct markdown) and more
 that could be useful for the developers in order to implement this engine.
-If you don't know what to write, let this part blank. -->
+If you don't know what to write, let this part blank.>
 
 **Applicable category of this engine**
 <!-- Where should this new engine fit in SearXNG? Current categories in SearXNG:

.github/dependabot.yml

@@ -1,4 +1,4 @@
-# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
 version: 2
 updates:
   - package-ecosystem: "pip"
@@ -8,9 +8,6 @@ updates:
       day: "friday"
     open-pull-requests-limit: 5
     target-branch: "master"
-    commit-message:
-      prefix: "[upd] pypi:"
-
   - package-ecosystem: "npm"
     directory: "/searx/static/themes/simple"
     schedule:
@@ -18,5 +15,3 @@ updates:
       day: "friday"
     open-pull-requests-limit: 5
     target-branch: "master"
-    commit-message:
-      prefix: "[upd] npm:"

.github/workflows/checker.yml

@@ -1,31 +0,0 @@
-name: "Checker"
-on:
-  schedule:
-    - cron: "0 4 * * 5"
-  workflow_dispatch:
-
-jobs:
-  checker:
-    name: Checker
-    runs-on: ubuntu-22.04
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
-      - name: Install Ubuntu packages
-        run: |
-          sudo ./utils/searxng.sh install packages
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: '3.11'
-          architecture: 'x64'
-
-      - name: Install Python dependencies
-        run: |
-          make V=1 install
-
-      - name: Checker
-        run: |
-          make search.checker

.github/workflows/data-update.yml

@@ -7,7 +7,7 @@ on:
 jobs:
   updateData:
     name: Update data - ${{ matrix.fetch }}
-    runs-on: ubuntu-24.04
+    runs-on: ubuntu-20.04
     if: ${{ github.repository_owner == 'searxng'}}
     strategy:
       fail-fast: false
@@ -22,16 +22,16 @@ jobs:
           - update_engine_descriptions.py
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@v2
 
       - name: Install Ubuntu packages
         run: |
           sudo ./utils/searxng.sh install packages
 
       - name: Set up Python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v2
         with:
-          python-version: '3.12'
+          python-version: '3.9'
           architecture: 'x64'
 
       - name: Install Python dependencies
@@ -46,18 +46,18 @@ jobs:
 
       - name: Create Pull Request
         id: cpr
-        uses: peter-evans/create-pull-request@v6
+        uses: peter-evans/create-pull-request@v3
         with:
-          commit-message: '[data] update searx.data - ${{ matrix.fetch }}'
+          commit-message: Update searx.data - ${{ matrix.fetch }}
           committer: searxng-bot <noreply@github.com>
           author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
           signoff: false
           branch: update_data_${{ matrix.fetch }}
           delete-branch: true
           draft: false
-          title: '[data] update searx.data - ${{ matrix.fetch }}'
+          title: 'Update searx.data - ${{ matrix.fetch }}'
           body: |
-            update searx.data - ${{ matrix.fetch }}
+            Update searx.data - ${{ matrix.fetch }}
           labels: |
             data
 

.github/workflows/integration.yml

@@ -16,16 +16,16 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-20.04]
-        python-version:  ["3.9", "3.10", "3.11", "3.12",]
+        python-version:  ["3.8", "3.9", "3.10", "3.11"]
     steps:
     - name: Checkout
-      uses: actions/checkout@v4
+      uses: actions/checkout@v2
     - name: Install Ubuntu packages
       run: |
         sudo ./utils/searxng.sh install packages
         sudo apt install firefox
     - name: Set up Python
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
         architecture: 'x64'
@@ -45,19 +45,27 @@ jobs:
         make V=1 gecko.driver
     - name: Run tests
       run: make V=1 ci.test
+    - name: Test coverage
+      run: make V=1 test.coverage
+    - name: Store coverage result
+      uses: actions/upload-artifact@v2
+      with:
+        name: coverage-${{ matrix.python-version }}
+        path: coverage/
+        retention-days: 60
 
   themes:
     name: Themes
     runs-on: ubuntu-20.04
     steps:
     - name: Checkout
-      uses: actions/checkout@v4
+      uses: actions/checkout@v2
     - name: Install Ubuntu packages
       run: sudo ./utils/searxng.sh install buildhost
     - name: Set up Python
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v4
       with:
-        python-version: '3.12'
+        python-version: '3.9'
         architecture: 'x64'
     - name: Cache Python dependencies
       id: cache-python
@@ -67,7 +75,7 @@ jobs:
           ./local
           ./.nvm
           ./node_modules
-        key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
+        key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
     - name: Install node dependencies
       run: make V=1 node.env
     - name: Build themes
@@ -80,16 +88,16 @@ jobs:
       contents: write  # for JamesIves/github-pages-deploy-action to push changes in repo
     steps:
     - name: Checkout
-      uses: actions/checkout@v4
+      uses: actions/checkout@v2
       with:
         fetch-depth: '0'
         persist-credentials: false
     - name: Install Ubuntu packages
       run: sudo ./utils/searxng.sh install buildhost
     - name: Set up Python
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v4
       with:
-        python-version: '3.12'
+        python-version: '3.9'
         architecture: 'x64'
     - name: Cache Python dependencies
       id: cache-python
@@ -99,7 +107,7 @@ jobs:
           ./local
           ./.nvm
           ./node_modules
-        key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
+        key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
     - name: Build documentation
       run: |
         make V=1 docs.clean docs.html
@@ -112,7 +120,7 @@ jobs:
         FOLDER: dist/docs
         CLEAN: true # Automatically remove deleted files from the deploy branch
         SINGLE_COMMIT: True
-        COMMIT_MESSAGE: '[doc] build from commit ${{ github.sha }}'
+        COMMIT_MESSAGE: build from commit ${{ github.sha }}
 
   babel:
     name: Update translations branch
@@ -126,14 +134,14 @@ jobs:
       contents: write  # for make V=1 weblate.push.translations
     steps:
     - name: Checkout
-      uses: actions/checkout@v4
+      uses: actions/checkout@v2
       with:
         fetch-depth: '0'
         token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
     - name: Set up Python
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v4
       with:
-        python-version: '3.12'
+        python-version: '3.9'
         architecture: 'x64'
     - name: Cache Python dependencies
       id: cache-python
@@ -143,7 +151,7 @@ jobs:
           ./local
           ./.nvm
           ./node_modules
-        key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
+        key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
     - name: weblate & git setup
       env:
         WEBLATE_CONFIG: ${{ secrets.WEBLATE_CONFIG }}
@@ -155,6 +163,7 @@ jobs:
     - name: Update transations
       id: update
       run: |
+        git restore utils/brand.env
         make V=1 weblate.push.translations
 
   dockers:
@@ -170,14 +179,14 @@ jobs:
     steps:
       - name: Checkout
         if: env.DOCKERHUB_USERNAME != null
-        uses: actions/checkout@v4
+        uses: actions/checkout@v2
         with:
           # make sure "make docker.push" can get the git history
           fetch-depth: '0'
       - name: Set up Python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v4
         with:
-          python-version: '3.12'
+          python-version: '3.9'
           architecture: 'x64'
       - name: Cache Python dependencies
         id: cache-python
@@ -187,7 +196,7 @@ jobs:
             ./local
             ./.nvm
             ./node_modules
-          key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
+          key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
       - name: Set up QEMU
         if: env.DOCKERHUB_USERNAME != null
         uses: docker/setup-qemu-action@v1

.github/workflows/security.yml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-20.04
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@v2
 
       - name: Run Trivy vulnerability scanner
         uses: aquasecurity/trivy-action@master

.github/workflows/translations-update.yml

@@ -6,19 +6,19 @@ on:
 
 jobs:
   babel:
-    name: "create PR for additions from weblate"
+    name: "create PR for additons from weblate"
     runs-on: ubuntu-20.04
     if: ${{ github.repository_owner == 'searxng' && github.ref == 'refs/heads/master' }}
     steps:
     - name: Checkout
-      uses: actions/checkout@v4
+      uses: actions/checkout@v2
       with:
         fetch-depth: '0'
         token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
     - name: Set up Python
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v4
       with:
-        python-version: '3.12'
+        python-version: '3.9'
         architecture: 'x64'
     - name: Cache Python dependencies
       id: cache-python
@@ -28,7 +28,7 @@ jobs:
           ./local
           ./.nvm
           ./node_modules
-        key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
+        key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
     - name: weblate & git setup
       env:
         WEBLATE_CONFIG: ${{ secrets.WEBLATE_CONFIG }}
@@ -45,15 +45,15 @@ jobs:
       uses: peter-evans/create-pull-request@v3
       with:
         token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
-        commit-message: '[l10n] update translations from Weblate'
+        commit-message: Update translations
         committer: searxng-bot <searxng-bot@users.noreply.github.com>
         author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
         signoff: false
         branch: translations_update
         delete-branch: true
         draft: false
-        title: '[l10n] update translations from Weblate'
+        title: 'Update translations'
         body: |
-          update translations from Weblate
+          Update translations
         labels: |
           translation

.gitignore

@@ -23,6 +23,3 @@ gh-pages/
 .idea/
 
 searx/version_frozen.py
-.dir-locals.el
-
-.python-version

.nvmrc

@@ -1 +1 @@
-v20.10
+v16.15.1

.pylintrc

@@ -1,4 +1,4 @@
-# -*- coding: utf-8; mode: conf-unix -*-
+# -*- coding: utf-8; mode: conf -*-
 # lint Python modules using external checkers.
 #
 # This is the main checker controlling the other ones and the reports
@@ -27,7 +27,7 @@ ignore-patterns=
 #init-hook=
 
 # Use multiple processes to speed up Pylint.
-jobs=0
+jobs=1
 
 # List of plugins (as comma separated values of python modules names) to load,
 # usually to register additional checkers.
@@ -338,7 +338,6 @@ valid-metaclass-classmethod-first-arg=mcs
 
 # Maximum number of arguments for function / method
 max-args=8
-max-positional-arguments=14
 
 # Maximum number of attributes for a class (see R0902).
 max-attributes=20

.tool-versions

@@ -1,2 +0,0 @@
-python 3.12.0
-shellcheck 0.9.0

AUTHORS.rst

@@ -168,9 +168,3 @@ features or generally made searx better:
 - Milad Laly @Milad-Laly
 - @llmII
 - @blob42 `<https://blob42.xyz>`_
-- Paolo Basso `<https://github.com/paolobasso99>`
-- Bernie Huang `<https://github.com/BernieHuang2008>`
-- Austin Olacsi `<https://github.com/Austin-Olacsi>`
-- @micsthepick
-- Daniel Kukula `<https://github.com/dkuku>`
-- Patrick Evans `https://github.com/holysoles`

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 ## Resources in the documentation
 
-* [Development quickstart](https://docs.searxng.org/dev/quickstart.html)
+* [Development quickstart](https://docs.searxng.org/dev/contribution_guide.html)
 * [Contribution guide](https://docs.searxng.org/dev/contribution_guide.html)
 
 ## Submitting PRs

Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.20
+FROM alpine:3.18
 ENTRYPOINT ["/sbin/tini","--","/usr/local/searxng/dockerfiles/docker-entrypoint.sh"]
 EXPOSE 8080
 VOLUME /etc/searxng
@@ -15,9 +15,7 @@ ENV INSTANCE_NAME=searxng \
     MORTY_KEY= \
     MORTY_URL= \
     SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml \
-    UWSGI_SETTINGS_PATH=/etc/searxng/uwsgi.ini \
-    UWSGI_WORKERS=%k \
-    UWSGI_THREADS=4
+    UWSGI_SETTINGS_PATH=/etc/searxng/uwsgi.ini
 
 WORKDIR /usr/local/searxng
 
@@ -35,6 +33,7 @@ RUN apk add --no-cache -t build-dependencies \
     git \
  && apk add --no-cache \
     ca-certificates \
+    su-exec \
     python3 \
     py3-pip \
     libxml2 \
@@ -44,7 +43,7 @@ RUN apk add --no-cache -t build-dependencies \
     uwsgi \
     uwsgi-python3 \
     brotli \
- && pip3 install --break-system-packages --no-cache -r requirements.txt \
+ && pip3 install --no-cache -r requirements.txt \
  && apk del build-dependencies \
  && rm -rf /root/.cache
 

Makefile

@@ -44,10 +44,10 @@ lxc.clean:
 
 PHONY += search.checker search.checker.%
 search.checker: install
-	$(Q)./manage pyenv.cmd searxng-checker -v
+	$(Q)./manage pyenv.cmd searx-checker -v
 
 search.checker.%: install
-	$(Q)./manage pyenv.cmd searxng-checker -v "$(subst _, ,$(patsubst search.checker.%,%,$@))"
+	$(Q)./manage pyenv.cmd searx-checker -v "$(subst _, ,$(patsubst search.checker.%,%,$@))"
 
 PHONY += test ci.test test.shell
 ci.test: test.yamllint test.black test.pyright test.pylint test.unit test.robot test.rst test.pybabel
@@ -56,32 +56,34 @@ test.shell:
 	$(Q)shellcheck -x -s dash \
 		dockerfiles/docker-entrypoint.sh
 	$(Q)shellcheck -x -s bash \
-		utils/brand.sh \
+		utils/brand.env \
 		$(MTOOLS) \
 		utils/lib.sh \
-		utils/lib_sxng*.sh \
-		utils/lib_go.sh \
 		utils/lib_nvm.sh \
+		utils/lib_static.sh \
+		utils/lib_go.sh \
 		utils/lib_redis.sh \
-		utils/searxng.sh \
-		utils/lxc.sh \
-		utils/lxc-searxng.env \
-		utils/searx.sh \
 		utils/filtron.sh \
-		utils/morty.sh
+		utils/searx.sh \
+		utils/searxng.sh \
+		utils/morty.sh \
+		utils/lxc.sh \
+		utils/lxc-searxng.env
 	$(Q)$(MTOOLS) build_msg TEST "$@ OK"
 
 
 # wrap ./manage script
 
+MANAGE += buildenv
 MANAGE += weblate.translations.commit weblate.push.translations
-MANAGE += data.all data.traits data.useragents data.locales
+MANAGE += data.all data.traits data.useragents
 MANAGE += docs.html docs.live docs.gh-pages docs.prebuild docs.clean
 MANAGE += docker.build docker.push docker.buildx
 MANAGE += gecko.driver
 MANAGE += node.env node.env.dev node.clean
 MANAGE += py.build py.clean
 MANAGE += pyenv pyenv.install pyenv.uninstall
+MANAGE += pypi.upload pypi.upload.test
 MANAGE += format.python
 MANAGE += test.yamllint test.pylint test.pyright test.black test.pybabel test.unit test.coverage test.robot test.rst test.clean
 MANAGE += themes.all themes.simple themes.simple.test pygments.less

PULL_REQUEST_TEMPLATE.md

@@ -12,11 +12,11 @@
 
 ## How to test this PR locally?
 
-<!-- commands to run the tests or instructions to test the changes -->
+<!-- commands to run the tests or instructions to test the changes-->
 
 ## Author's checklist
 
-<!-- additional notes for reviewers -->
+<!-- additional notes for reviewiers -->
 
 ## Related issues
 

README.rst

@@ -66,7 +66,7 @@ A user_, admin_ and developer_ handbook is available on the homepage_.
 Contact
 =======
 
-Ask questions or chat with the SearXNG community (this not a chatbot) on
+Ask questions or just chat about SearXNG on
 
 IRC
   `#searxng on libera.chat <https://web.libera.chat/?channel=#searxng>`_
@@ -75,42 +75,76 @@ IRC
 Matrix
   `#searxng:matrix.org <https://matrix.to/#/#searxng:matrix.org>`_
 
+Differences to searx
+====================
+
+SearXNG is a fork of `searx`_, with notable changes:
+
+.. _searx: https://github.com/searx/searx
+
+
+User experience
+---------------
+
+- Reworked (and still simple) theme:
+
+  * Usable on desktop, tablet and mobile.
+  * Light and dark versions (available in the preferences).
+  * Right-to-left language support.
+  * `Screenshots <https://dev.searxng.org/screenshots.html>`_
+
+- The translations are up to date, you can contribute on `Weblate`_
+- The preferences page has been updated:
+
+  * Browse which engines are reliable or not.
+  * Engines are grouped inside each tab.
+  * Each engine has a description.
+
+- Thanks to the anonymous metrics, it is easier to report malfunctioning engines,
+  so they get fixed quicker
+
+  - `Turn off metrics on the server
+    <https://docs.searxng.org/admin/engines/settings.html#general>`_ if you don't want them recorded.
+
+- Administrators can `block and/or replace the URLs in the search results
+  <https://github.com/searxng/searxng/blob/5c1c0817c3996c5670a545d05831d234d21e6217/searx/settings.yml#L191-L199>`_
+
 
 Setup
-=====
+-----
 
-- A well maintained `Docker image`_, also built for ARM64 and ARM/v7
-  architectures.
-- Alternatively there are *up to date* `installation scripts`_.
-- For individual setup consult our detailed `Step by step`_ instructions.
-- To fine-tune your instance, take a look at the `Administrator documentation`_.
+- No need for `Morty`_ to proxy images, even on a public instance.
+- No need for `Filtron`_ to block bots, as there is now a built-in `limiter`_.
+- A well maintained `Docker image`_, now also built for ARM64 and ARM/v7 architectures.
+  (Alternatively there are up to date installation scripts.)
 
-.. _Administrator documentation: https://docs.searxng.org/admin/index.html
-.. _Step by step: https://docs.searxng.org/admin/installation-searxng.html
-.. _installation scripts: https://docs.searxng.org/admin/installation-scripts.html
 .. _Docker image: https://github.com/searxng/searxng-docker
 
+
+Contributing
+------------
+
+- Readable debug log.
+- Contributing is easier, thanks to the `Development Quickstart`_ guide.
+- A lot of code cleanup and bugfixes.
+- Up to date list dependencies.
+
+.. _Morty: https://github.com/asciimoo/morty
+.. _Filtron: https://github.com/searxng/filtron
+.. _limiter: https://docs.searxng.org/src/searx.plugins.limiter.html
+.. _Weblate: https://translate.codeberg.org/projects/searxng/searxng/
+.. _Development Quickstart: https://docs.searxng.org/dev/quickstart.html
+
+
 Translations
 ============
 
-.. _Weblate: https://translate.codeberg.org/projects/searxng/searxng/
-
 Help translate SearXNG at `Weblate`_
 
 .. figure:: https://translate.codeberg.org/widgets/searxng/-/multi-auto.svg
    :target: https://translate.codeberg.org/projects/searxng/
 
 
-Contributing
-============
-
-.. _development quickstart: https://docs.searxng.org/dev/quickstart.html
-.. _developer documentation: https://docs.searxng.org/dev/index.html
-
-Are you a developer?  Have a look at our `development quickstart`_ guide, it's
-very easy to contribute.  Additionally we have a `developer documentation`_.
-
-
 Codespaces
 ==========
 
@@ -121,9 +155,9 @@ You can contribute from your browser using `GitHub Codespaces`_:
 - Click on the ``Codespaces`` tab instead of ``Local``
 - Click on ``Create codespace on master``
 - VSCode is going to start in the browser
-- Wait for ``git pull && make install`` to appear and then disappear
+- Wait for ``git pull && make install`` to appears and then to disapear
 - You have `120 hours per month`_ (see also your `list of existing Codespaces`_)
-- You can start SearXNG using ``make run`` in the terminal or by pressing ``Ctrl+Shift+B``
+- You can start SearXNG using ``make run`` in the terminal or by pressing ``Ctrl+Shift+B``.
 
 .. _GitHub Codespaces: https://docs.github.com/en/codespaces/overview
 .. _120 hours per month: https://github.com/settings/billing

dockerfiles/docker-entrypoint.sh

@@ -175,4 +175,4 @@ unset MORTY_KEY
 
 # Start uwsgi
 printf 'Listen on %s\n' "${BIND_ADDRESS}"
-exec uwsgi --master --uid searxng --gid searxng --http-socket "${BIND_ADDRESS}" "${UWSGI_SETTINGS_PATH}"
+exec su-exec searxng:searxng uwsgi --master --http-socket "${BIND_ADDRESS}" "${UWSGI_SETTINGS_PATH}"

dockerfiles/uwsgi.ini

@@ -4,12 +4,8 @@ uid = searxng
 gid = searxng
 
 # Number of workers (usually CPU count)
-# default value: %k (= number of CPU core, see Dockerfile)
-workers = $(UWSGI_WORKERS)
-
-# Number of threads per worker
-# default value: 4 (see Dockerfile)
-threads = $(UWSGI_THREADS)
+workers = %k
+threads = 4
 
 # The right granted on the created socket
 chmod-socket = 666
@@ -42,13 +38,12 @@ buffer-size = 8192
 # See https://github.com/searx/searx-docker/issues/24
 add-header = Connection: close
 
-# Follow SIGTERM convention
-# See https://github.com/searxng/searxng/issues/3427
-die-on-term
-
 # uwsgi serves the static files
+# expires set to one year since there are hashes
 static-map = /static=/usr/local/searxng/searx/static
-# expires set to one day
-static-expires = /* 86400
+static-expires = /* 31557600
 static-gzip-all = True
 offload-threads = %k
+
+# Cache
+cache2 = name=searxngcache,items=2000,blocks=2000,blocksize=4096,bitmap=1

docs/_themes/searxng/theme.conf

@@ -4,4 +4,3 @@ stylesheet = searxng.css
 
 [options]
 touch_icon =
-globaltoc_maxdepth = 5

docs/admin/api.rst

@@ -69,6 +69,10 @@ Sample response
        {
          "enabled": true,
          "name": "HTTPS rewrite"
+       },
+       {
+         "enabled": false,
+         "name": "Vim-like hotkeys"
        }
      ],
      "safe_search": 0
@@ -84,9 +88,9 @@ HTML of the site.  URL of the SearXNG instance and values are customizable.
 .. code:: html
 
    <form method="post" action="https://example.org/">
-     <!-- search      --> <input type="text" name="q">
-     <!-- categories  --> <input type="hidden" name="categories" value="general,social media">
-     <!-- language    --> <input type="hidden" name="lang" value="all">
-     <!-- locale      --> <input type="hidden" name="locale" value="en">
-     <!-- date filter --> <input type="hidden" name="time_range" value="month">
+     <!-- search      --> <input type="text" name="q" />
+     <!-- categories  --> <input type="hidden" name="categories" value="general,social media" />
+     <!-- language    --> <input type="hidden" name="lang" value="all" />
+     <!-- locale      --> <input type="hidden" name="locale" value="en" />
+     <!-- date filter --> <input type="hidden" name="time_range" value="month" />
    </form>

docs/admin/buildhosts.rst

@@ -4,36 +4,28 @@
 Buildhosts
 ==========
 
-.. contents::
+.. sidebar:: This article needs some work
+
+   If you have any contribution send us your :pull:`PR <../pulls>`, see
+   :ref:`how to contribute`.
+
+.. contents:: Contents
    :depth: 2
    :local:
    :backlinks: entry
 
-To get best results from build, it's recommend to install additional packages on
-build hosts (see :ref:`searxng.sh`).
+To get best results from build, it's recommend to install additional packages
+on build hosts (see :ref:`searxng.sh`).::
 
-.. _searxng.sh install buildhost:
+  sudo -H ./utils/searxng.sh install buildhost
 
-Build and Development tools
-===========================
-
-To Install tools used by build and development tasks in once:
-
-.. tabs::
-
-  .. group-tab:: SearXNG's development tools
-
-     .. code:: sh
-
-        $ sudo -H ./utils/searxng.sh install buildhost
-
-This will install packages needed by SearXNG:
+This will install packages needed by searx:
 
 .. kernel-include:: $DOCS_BUILD/includes/searxng.rst
    :start-after: START distro-packages
    :end-before: END distro-packages
 
-and packages needed to build documentation and run tests:
+and packages needed to build docuemtation and run tests:
 
 .. kernel-include:: $DOCS_BUILD/includes/searxng.rst
    :start-after: START build-packages
@@ -81,7 +73,7 @@ If your docs build (``make docs.html``) shows warnings like this::
             display), check the imgmath_latex setting
 
 you need to install additional packages on your build host, to get better HTML
-output (:ref:`install buildhost <searxng.sh install buildhost>`).
+output.
 
 .. tabs::
 
@@ -101,7 +93,7 @@ output (:ref:`install buildhost <searxng.sh install buildhost>`).
 
       .. code-block:: sh
 
-         $ sudo dnf install graphviz graphviz-gd ImageMagick texlive-xetex-bin librsvg2-tools
+         $ sudo dnf install graphviz graphviz-gd texlive-xetex-bin librsvg2-tools
 
 
 For PDF output you also need:
@@ -126,7 +118,8 @@ For PDF output you also need:
 
       	 $ sudo dnf install \
 	        texlive-collection-fontsrecommended texlive-collection-latex \
-             dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts
+		dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts \
+		ImageMagick
 
 .. _sh lint:
 
@@ -135,8 +128,7 @@ Lint shell scripts
 
 .. _ShellCheck: https://github.com/koalaman/shellcheck
 
-To lint shell scripts we use ShellCheck_ - a shell script static analysis tool
-(:ref:`install buildhost <searxng.sh install buildhost>`).
+To lint shell scripts, we use ShellCheck_ - a shell script static analysis tool.
 
 .. SNIP sh lint requirements
 

docs/admin/engines/command-line-engines.rst

@@ -0,0 +1,79 @@
+.. _engine command:
+
+====================
+Command Line Engines
+====================
+
+.. sidebar:: info
+
+   - :origin:`command.py <searx/engines/command.py>`
+   - :ref:`offline engines`
+
+With *command engines* administrators can run engines to integrate arbitrary
+shell commands.
+
+When creating and enabling a ``command`` engine on a public instance, you must
+be careful to avoid leaking private data.  The easiest solution is to limit the
+access by setting ``tokens`` as described in section :ref:`private engines`.
+
+The engine base is flexible.  Only your imagination can limit the power of this
+engine (and maybe security concerns).  The following options are available:
+
+``command``:
+  A comma separated list of the elements of the command.  A special token
+  ``{{QUERY}}`` tells where to put the search terms of the user. Example:
+
+  .. code:: yaml
+
+     ['ls', '-l', '-h', '{{QUERY}}']
+
+``delimiter``:
+  A mapping containing a delimiter ``char`` and the *titles* of each element in
+  ``keys``.
+
+``parse_regex``:
+  A dict containing the regular expressions for each result key.
+
+``query_type``:
+
+  The expected type of user search terms.  Possible values: ``path`` and
+  ``enum``.
+
+  ``path``:
+    Checks if the user provided path is inside the working directory.  If not,
+    the query is not executed.
+
+  ``enum``:
+    Is a list of allowed search terms.  If the user submits something which is
+    not included in the list, the query returns an error.
+
+``query_enum``:
+  A list containing allowed search terms if ``query_type`` is set to ``enum``.
+
+``working_dir``:
+
+  The directory where the command has to be executed.  Default: ``./``
+
+``result_separator``:
+  The character that separates results. Default: ``\n``
+
+The example engine below can be used to find files with a specific name in the
+configured working directory:
+
+.. code:: yaml
+
+  - name: find
+    engine: command
+    command: ['find', '.', '-name', '{{QUERY}}']
+    query_type: path
+    shortcut: fnd
+    delimiter:
+        chars: ' '
+        keys: ['line']
+
+
+Acknowledgment
+==============
+
+This development was sponsored by `Search and Discovery Fund
+<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

docs/admin/engines/index.rst

@@ -0,0 +1,26 @@
+.. _engines and settings:
+
+==================
+Engines & Settings
+==================
+
+.. sidebar:: Further reading ..
+
+   - :ref:`settings engine`
+   - :ref:`engine settings` & :ref:`engine file`
+
+.. toctree::
+   :maxdepth: 3
+
+   settings
+
+.. toctree::
+   :maxdepth: 1
+
+   private-engines
+   recoll
+   sql-engines
+   nosql-engines
+   search-indexer-engines
+   command-line-engines
+   searx.engines.xpath

docs/admin/engines/nosql-engines.rst

@@ -1,5 +1,3 @@
-.. _nosql engines:
-
 ===============
 NoSQL databases
 ===============
@@ -10,16 +8,6 @@ NoSQL databases
    - `redis.io <https://redis.io/>`_
    - `MongoDB <https://www.mongodb.com>`_
 
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. sidebar:: info
-
-   Initial sponsored by `Search and Discovery Fund
-   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
-
 The following `NoSQL databases`_ are supported:
 
 - :ref:`engine redis_server`
@@ -42,8 +30,15 @@ can still add them and limit the access by setting ``tokens`` as described in
 section :ref:`private engines`.
 
 
+Configure the engines
+=====================
+
+`NoSQL databases`_ are used for storing arbitrary data without first defining
+their structure.
+
+
 Extra Dependencies
-==================
+------------------
 
 For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to
 install additional packages in Python's Virtual Environment of your SearXNG
@@ -54,13 +49,6 @@ instance.  To switch into the environment (:ref:`searxng-src`) you can use
   (searxng-pyenv)$ pip install ...
 
 
-Configure the engines
-=====================
-
-`NoSQL databases`_ are used for storing arbitrary data without first defining
-their structure.
-
-
 .. _engine redis_server:
 
 Redis Server
@@ -74,9 +62,29 @@ Redis Server
    - redis.io_
    - :origin:`redis_server.py <searx/engines/redis_server.py>`
 
-.. automodule:: searx.engines.redis_server
-  :members:
 
+Redis is an open source (BSD licensed), in-memory data structure (key value
+based) store.  Before configuring the ``redis_server`` engine, you must install
+the dependency redis_.
+
+Select a database to search in and set its index in the option ``db``.  You can
+either look for exact matches or use partial keywords to find what you are
+looking for by configuring ``exact_match_only``.  You find an example
+configuration below:
+
+.. code:: yaml
+
+  # Required dependency: redis
+
+  - name: myredis
+    shortcut : rds
+    engine: redis_server
+    exact_match_only: false
+    host: '127.0.0.1'
+    port: 6379
+    enable_http: true
+    password: ''
+    db: 0
 
 .. _engine mongodb:
 
@@ -91,7 +99,37 @@ MongoDB
    - MongoDB_
    - :origin:`mongodb.py <searx/engines/mongodb.py>`
 
+MongoDB_ is a document based database program that handles JSON like data.
+Before configuring the ``mongodb`` engine, you must install the dependency
+redis_.
 
-.. automodule:: searx.engines.mongodb
-  :members:
+In order to query MongoDB_, you have to select a ``database`` and a
+``collection``.  Furthermore, you have to select a ``key`` that is going to be
+searched.  MongoDB_ also supports the option ``exact_match_only``, so configure
+it as you wish.  Below is an example configuration for using a MongoDB
+collection:
+
+.. code:: yaml
+
+  # MongoDB engine
+  # Required dependency: pymongo
+
+  - name: mymongo
+    engine: mongodb
+    shortcut: md
+    exact_match_only: false
+    host: '127.0.0.1'
+    port: 27017
+    enable_http: true
+    results_per_page: 20
+    database: 'business'
+    collection: 'reviews'  # name of the db collection
+    key: 'name'            # key in the collection to search for
+
+
+Acknowledgment
+==============
+
+This development was sponsored by `Search and Discovery Fund
+<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
 

docs/admin/engines/private-engines.rst

@@ -0,0 +1,49 @@
+.. _private engines:
+
+============================
+Private Engines (``tokens``)
+============================
+
+Administrators might find themselves wanting to limit access to some of the
+enabled engines on their instances. It might be because they do not want to
+expose some private information through :ref:`offline engines`.  Or they would
+rather share engines only with their trusted friends or colleagues.
+
+To solve this issue the concept of *private engines* exists.
+
+
+A new option was added to engines named `tokens`. It expects a list of
+strings. If the user making a request presents one of the tokens of an engine,
+they can access information about the engine and make search requests.
+
+Example configuration to restrict access to the Arch Linux Wiki engine:
+
+.. code:: yaml
+
+  - name: arch linux wiki
+    engine: archlinux
+    shortcut: al
+    tokens: [ 'my-secret-token' ]
+
+
+Unless a user has configured the right token, the engine is going
+to be hidden from him/her. It is not going to be included in the
+list of engines on the Preferences page and in the output of
+`/config` REST API call.
+
+Tokens can be added to one's configuration on the Preferences page
+under "Engine tokens". The input expects a comma separated list of
+strings.
+
+The distribution of the tokens from the administrator to the users
+is not carved in stone. As providing access to such engines
+implies that the admin knows and trusts the user, we do not see
+necessary to come up with a strict process. Instead,
+we would like to add guidelines to the documentation of the feature.
+
+
+Acknowledgment
+==============
+
+This development was sponsored by `Search and Discovery Fund
+<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

docs/admin/engines/recoll.rst

@@ -0,0 +1,50 @@
+.. _engine recoll:
+
+=============
+Recoll Engine
+=============
+
+.. sidebar:: info
+
+   - `Recoll <https://www.lesbonscomptes.com/recoll/>`_
+   - `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_
+   - :origin:`searx/engines/recoll.py`
+
+Recoll_ is a desktop full-text search tool based on Xapian.  By itself Recoll_
+does not offer WEB or API access, this can be achieved using recoll-webui_
+
+
+Configuration
+=============
+
+You must configure the following settings:
+
+``base_url``:
+  Location where recoll-webui can be reached.
+
+``mount_prefix``:
+  Location where the file hierarchy is mounted on your *local* filesystem.
+
+``dl_prefix``:
+  Location where the file hierarchy as indexed by recoll can be reached.
+
+``search_dir``:
+  Part of the indexed file hierarchy to be search, if empty the full domain is
+  searched.
+
+
+Example
+=======
+
+Scenario:
+
+#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``,
+#. the Recoll search interface can be reached at https://recoll.example.org/ and
+#. the contents of this filesystem can be reached though https://download.example.org/reference
+
+.. code:: yaml
+
+   base_url: https://recoll.example.org/
+   mount_prefix: /export/documents
+   dl_prefix: https://download.example.org
+   search_dir: ''

docs/admin/engines/search-indexer-engines.rst

@@ -0,0 +1,136 @@
+====================
+Local Search Engines
+====================
+
+.. sidebar:: further read
+
+   - `Comparison to alternatives
+     <https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
+
+Administrators might find themselves wanting to integrate locally running search
+engines.  The following ones are supported for now:
+
+* `Elasticsearch`_
+* `Meilisearch`_
+* `Solr`_
+
+Each search engine is powerful, capable of full-text search.  All of the engines
+above are added to ``settings.yml`` just commented out, as you have to
+``base_url`` for all them.
+
+Please note that if you are not using HTTPS to access these engines, you have to enable
+HTTP requests by setting ``enable_http`` to ``True``.
+
+Furthermore, if you do not want to expose these engines on a public instance, you
+can still add them and limit the access by setting ``tokens`` as described in
+section :ref:`private engines`.
+
+.. _engine meilisearch:
+
+MeiliSearch
+===========
+
+.. sidebar:: info
+
+   - :origin:`meilisearch.py <searx/engines/meilisearch.py>`
+   - `MeiliSearch <https://www.meilisearch.com>`_
+   - `MeiliSearch Documentation <https://docs.meilisearch.com/>`_
+   - `Install MeiliSearch
+     <https://docs.meilisearch.com/learn/getting_started/installation.html>`_
+
+MeiliSearch_ is aimed at individuals and small companies.  It is designed for
+small-scale (less than 10 million documents) data collections.  E.g. it is great
+for storing web pages you have visited and searching in the contents later.
+
+The engine supports faceted search, so you can search in a subset of documents
+of the collection.  Furthermore, you can search in MeiliSearch_ instances that
+require authentication by setting ``auth_token``.
+
+Here is a simple example to query a Meilisearch instance:
+
+.. code:: yaml
+
+  - name: meilisearch
+    engine: meilisearch
+    shortcut: mes
+    base_url: http://localhost:7700
+    index: my-index
+    enable_http: true
+
+
+.. _engine elasticsearch:
+
+Elasticsearch
+=============
+
+.. sidebar:: info
+
+   - :origin:`elasticsearch.py <searx/engines/elasticsearch.py>`
+   - `Elasticsearch <https://www.elastic.co/elasticsearch/>`_
+   - `Elasticsearch Guide
+     <https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_
+   - `Install Elasticsearch
+     <https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_
+
+Elasticsearch_ supports numerous ways to query the data it is storing.  At the
+moment the engine supports the most popular search methods (``query_type``):
+
+- ``match``,
+- ``simple_query_string``,
+- ``term`` and
+- ``terms``.
+
+If none of the methods fit your use case, you can select ``custom`` query type
+and provide the JSON payload to submit to Elasticsearch in
+``custom_query_json``.
+
+The following is an example configuration for an Elasticsearch_ instance with
+authentication configured to read from ``my-index`` index.
+
+.. code:: yaml
+
+  - name: elasticsearch
+    shortcut: es
+    engine: elasticsearch
+    base_url: http://localhost:9200
+    username: elastic
+    password: changeme
+    index: my-index
+    query_type: match
+    # custom_query_json: '{ ... }'
+    enable_http: true
+
+.. _engine solr:
+
+Solr
+====
+
+.. sidebar:: info
+
+   - :origin:`solr.py <searx/engines/solr.py>`
+   - `Solr <https://solr.apache.org>`_
+   - `Solr Resources <https://solr.apache.org/resources.html>`_
+   - `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_
+
+Solr_ is a popular search engine based on Lucene, just like Elasticsearch_.  But
+instead of searching in indices, you can search in collections.
+
+This is an example configuration for searching in the collection
+``my-collection`` and get the results in ascending order.
+
+.. code:: yaml
+
+  - name: solr
+    engine: solr
+    shortcut: slr
+    base_url: http://localhost:8983
+    collection: my-collection
+    sort: asc
+    enable_http: true
+
+
+Acknowledgment
+==============
+
+This development was sponsored by `Search and Discovery Fund
+<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

docs/admin/engines/searx.engines.xpath.rst

@@ -4,11 +4,6 @@
 XPath Engine
 ============
 
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
 .. automodule:: searx.engines.xpath
   :members:
 

docs/admin/engines/settings.rst

@@ -0,0 +1,732 @@
+.. _settings.yml:
+
+================
+``settings.yml``
+================
+
+This page describe the options possibilities of the :origin:`searx/settings.yml`
+file.
+
+.. sidebar:: Further reading ..
+
+   - :ref:`use_default_settings.yml`
+   - :ref:`search API`
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+.. _settings location:
+
+settings.yml location
+=====================
+
+The initial ``settings.yml`` we be load from these locations:
+
+1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
+2. ``/etc/searxng/settings.yml``
+
+If these files don't exist (or are empty or can't be read), SearXNG uses the
+:origin:`searx/settings.yml` file.  Read :ref:`settings use_default_settings` to
+see how you can simplify your *user defined* ``settings.yml``.
+
+
+.. _settings global:
+
+Global Settings
+===============
+
+.. _settings brand:
+
+``brand:``
+----------
+
+.. code:: yaml
+
+   brand:
+     issue_url: https://github.com/searxng/searxng/issues
+     docs_url: https://docs.searxng.org
+     public_instances: https://searx.space
+     wiki_url: https://github.com/searxng/searxng/wiki
+
+``issue_url`` :
+  If you host your own issue tracker change this URL.
+
+``docs_url`` :
+  If you host your own documentation change this URL.
+
+``public_instances`` :
+  If you host your own https://searx.space change this URL.
+
+``wiki_url`` :
+  Link to your wiki (or ``false``)
+
+.. _settings general:
+
+``general:``
+------------
+
+.. code:: yaml
+
+   general:
+     debug: false
+     instance_name:  "SearXNG"
+     privacypolicy_url: false
+     donation_url: false
+     contact_url: false
+     enable_metrics: true
+
+``debug`` : ``$SEARXNG_DEBUG``
+  Allow a more detailed log if you run SearXNG directly. Display *detailed* error
+  messages in the browser too, so this must be deactivated in production.
+
+``donation_url`` :
+  Set value to ``true`` to use your own donation page written in the
+  :ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable
+  the donation link altogether.
+
+``privacypolicy_url``:
+  Link to privacy policy.
+
+``contact_url``:
+  Contact ``mailto:`` address or WEB form.
+
+``enable_metrics``:
+  Enabled by default. Record various anonymous metrics availabled at ``/stats``,
+  ``/stats/errors`` and ``/preferences``.
+
+.. _settings search:
+
+``search:``
+-----------
+
+.. code:: yaml
+
+   search:
+     safe_search: 0
+     autocomplete: ""
+     default_lang: ""
+     ban_time_on_fail: 5
+     max_ban_time_on_fail: 120
+     suspended_times:
+       SearxEngineAccessDenied: 86400
+       SearxEngineCaptcha: 86400
+       SearxEngineTooManyRequests: 3600
+       cf_SearxEngineCaptcha: 1296000
+       cf_SearxEngineAccessDenied: 86400
+       recaptcha_SearxEngineCaptcha: 604800
+     formats:
+       - html
+
+``safe_search``:
+  Filter results.
+
+  - ``0``: None
+  - ``1``: Moderate
+  - ``2``: Strict
+
+``autocomplete``:
+  Existing autocomplete backends, leave blank to turn it off.
+
+  - ``dbpedia``
+  - ``duckduckgo``
+  - ``google``
+  - ``startpage``
+  - ``swisscows``
+  - ``qwant``
+  - ``wikipedia``
+
+``default_lang``:
+  Default search language - leave blank to detect from browser information or
+  use codes from :origin:`searx/languages.py`.
+
+``languages``:
+  List of available languages - leave unset to use all codes from
+  :origin:`searx/languages.py`.  Otherwise list codes of available languages.
+  The ``all`` value is shown as the ``Default language`` in the user interface
+  (in most cases, it is meant to send the query without a language parameter ;
+  in some cases, it means the English language) Example:
+
+  .. code:: yaml
+
+     languages:
+       - all
+       - en
+       - en-US
+       - de
+       - it-IT
+       - fr
+       - fr-BE
+
+``ban_time_on_fail``:
+  Ban time in seconds after engine errors.
+
+``max_ban_time_on_fail``:
+  Max ban time in seconds after engine errors.
+
+``suspended_times``:
+  Engine suspension time after error (in seconds; set to 0 to disable)
+
+  ``SearxEngineAccessDenied``: 86400
+    For error "Access denied" and "HTTP error [402, 403]"
+
+  ``SearxEngineCaptcha``: 86400
+    For error "CAPTCHA"
+
+  ``SearxEngineTooManyRequests``: 3600
+    For error "Too many request" and "HTTP error 429"
+
+  Cloudflare CAPTCHA:
+     - ``cf_SearxEngineCaptcha``: 1296000
+     - ``cf_SearxEngineAccessDenied``: 86400
+
+  Google CAPTCHA:
+    - ``recaptcha_SearxEngineCaptcha``: 604800
+
+``formats``:
+  Result formats available from web, remove format to deny access (use lower
+  case).
+
+  - ``html``
+  - ``csv``
+  - ``json``
+  - ``rss``
+
+
+.. _settings server:
+
+``server:``
+-----------
+
+.. code:: yaml
+
+   server:
+       base_url: http://example.org/location  # change this!
+       port: 8888
+       bind_address: "127.0.0.1"
+       secret_key: "ultrasecretkey"           # change this!
+       limiter: false
+       image_proxy: false
+       default_http_headers:
+         X-Content-Type-Options : nosniff
+         X-XSS-Protection : 1; mode=block
+         X-Download-Options : noopen
+         X-Robots-Tag : noindex, nofollow
+         Referrer-Policy : no-referrer
+
+
+``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>`
+  The base URL where SearXNG is deployed.  Used to create correct inbound links.
+  If you change the value, don't forget to rebuild instance's environment
+  (:ref:`utils/brand.env <make buildenv>`)
+
+``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>`
+  Port number and *bind address* of the SearXNG web application if you run it
+  directly using ``python searx/webapp.py``.  Doesn't apply to a SearXNG
+  services running behind a proxy and using socket communications.  If you
+  change the value, don't forget to rebuild instance's environment
+  (:ref:`utils/brand.env <make buildenv>`)
+
+``secret_key`` : ``$SEARXNG_SECRET``
+  Used for cryptography purpose.
+
+.. _limiter:
+
+``limiter`` :
+  Rate limit the number of request on the instance, block some bots.  The
+  :ref:`limiter src` requires a :ref:`settings redis` database.
+
+.. _image_proxy:
+
+``image_proxy`` :
+  Allow your instance of SearXNG of being able to proxy images.  Uses memory space.
+
+.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
+
+``default_http_headers`` :
+  Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
+
+
+.. _settings ui:
+
+``ui:``
+-------
+
+.. _cache busting:
+   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
+
+.. code:: yaml
+
+   ui:
+     static_use_hash: false
+     default_locale: ""
+     query_in_title: false
+     infinite_scroll: false
+     center_alignment: false
+     cache_url: https://web.archive.org/web/
+     default_theme: simple
+     theme_args:
+       simple_style: auto
+
+.. _static_use_hash:
+
+``static_use_hash`` :
+  Enables `cache busting`_ of static files.
+
+``default_locale`` :
+  SearXNG interface language.  If blank, the locale is detected by using the
+  browser language.  If it doesn't work, or you are deploying a language
+  specific instance of searx, a locale can be defined using an ISO language
+  code, like ``fr``, ``en``, ``de``.
+
+``query_in_title`` :
+  When true, the result page's titles contains the query it decreases the
+  privacy, since the browser can records the page titles.
+
+``infinite_scroll``:
+  When true, automatically loads the next page when scrolling to bottom of the current page.
+
+``center_alignment`` : default ``false``
+  When enabled, the results are centered instead of being in the left (or RTL)
+  side of the screen.  This setting only affects the *desktop layout*
+  (:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
+
+.. cache_url:
+
+``cache_url`` : ``https://web.archive.org/web/``
+  URL prefix of the internet archive or cache, don't forgett trailing slash (if
+  needed).  The default is https://web.archive.org/web/ alternatives are:
+
+  - https://webcache.googleusercontent.com/search?q=cache:
+  - https://archive.today/
+
+``default_theme`` :
+  Name of the theme you want to use by default on your SearXNG instance.
+
+``theme_args.simple_style``:
+  Style of simple theme: ``auto``, ``light``, ``dark``
+
+``results_on_new_tab``:
+  Open result links in a new tab by default.
+
+.. _settings redis:
+
+``redis:``
+----------
+
+.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
+
+A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
+will find a description to test your redis connection in SerXNG.  When using
+sockets, don't forget to check the access rights on the socket::
+
+  ls -la /usr/local/searxng-redis/run/redis.sock
+  srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
+
+In this example read/write access is given to the *searxng-redis* group.  To get
+access rights to redis instance (the socket), your SearXNG (or even your
+developer) account needs to be added to the *searxng-redis* group.
+
+``url``
+  URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
+
+    redis://[[username]:[password]]@localhost:6379/0
+    rediss://[[username]:[password]]@localhost:6379/0
+    unix://[[username]:[password]]@/path/to/socket.sock?db=0
+
+.. admonition:: Tip for developers
+
+   To set up a local redis instance, first set the socket path of the Redis DB
+   in your YAML setting:
+
+   .. code:: yaml
+
+      redis:
+        url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
+
+   Then use the following commands to install the redis instance ::
+
+     $ ./manage redis.build
+     $ sudo -H ./manage redis.install
+     $ sudo -H ./manage redis.addgrp "${USER}"
+     # don't forget to logout & login to get member of group
+
+
+.. _settings outgoing:
+
+``outgoing:``
+-------------
+
+Communication with search engines.
+
+.. code:: yaml
+
+   outgoing:
+     request_timeout: 2.0       # default timeout in seconds, can be override by engine
+     max_request_timeout: 10.0  # the maximum timeout in seconds
+     useragent_suffix: ""       # information like an email address to the administrator
+     pool_connections: 100      # Maximum number of allowable connections, or null
+                                # for no limits. The default is 100.
+     pool_maxsize: 10           # Number of allowable keep-alive connections, or null
+                                # to always allow. The default is 10.
+     enable_http2: true         # See https://www.python-httpx.org/http2/
+     # uncomment below section if you want to use a custom server certificate
+     # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
+     # and https://www.python-httpx.org/compatibility/#ssl-configuration
+     #  verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
+     #
+     # uncomment below section if you want to use a proxyq see: SOCKS proxies
+     #   https://2.python-requests.org/en/latest/user/advanced/#proxies
+     # are also supported: see
+     #   https://2.python-requests.org/en/latest/user/advanced/#socks
+     #
+     #  proxies:
+     #    all://:
+     #      - http://proxy1:8080
+     #      - http://proxy2:8080
+     #
+     #  using_tor_proxy: true
+     #
+     # Extra seconds to add in order to account for the time taken by the proxy
+     #
+     #  extra_proxy_timeout: 10.0
+     #
+
+``request_timeout`` :
+  Global timeout of the requests made to others engines in seconds.  A bigger
+  timeout will allow to wait for answers from slow engines, but in consequence
+  will slow SearXNG reactivity (the result page may take the time specified in the
+  timeout to load). Can be override by :ref:`settings engine`
+
+``useragent_suffix`` :
+  Suffix to the user-agent SearXNG uses to send requests to others engines.  If an
+  engine wish to block you, a contact info here may be useful to avoid that.
+
+``keepalive_expiry`` :
+  Number of seconds to keep a connection in the pool. By default 5.0 seconds.
+
+.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
+
+``proxies`` :
+  Define one or more proxies you wish to use, see `httpx proxies`_.
+  If there are more than one proxy for one protocol (http, https),
+  requests to the engines are distributed in a round-robin fashion.
+
+``source_ips`` :
+  If you use multiple network interfaces, define from which IP the requests must
+  be made. Example:
+
+  * ``0.0.0.0`` any local IPv4 address.
+  * ``::`` any local IPv6 address.
+  * ``192.168.0.1``
+  * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
+  * ``fe80::60a2:1691:e5a2:ee1f``
+  * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
+  * ``[ 192.168.0.1, fe80::/126 ]``
+
+``retries`` :
+  Number of retry in case of an HTTP error.  On each retry, SearXNG uses an
+  different proxy and source ip.
+
+``retry_on_http_error`` :
+  Retry request on some HTTP status code.
+
+  Example:
+
+  * ``true`` : on HTTP status code between 400 and 599.
+  * ``403`` : on HTTP status code 403.
+  * ``[403, 429]``: on HTTP status code 403 and 429.
+
+``enable_http2`` :
+  Enable by default. Set to ``false`` to disable HTTP/2.
+
+.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
+.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
+
+``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
+  Allow to specify a path to certificate.
+  see `httpx verification defaults`_.
+
+  In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
+  ``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
+  see `httpx ssl configuration`_.
+
+``max_redirects`` :
+  30 by default. Maximum redirect before it is an error.
+
+
+.. _settings categories_as_tabs:
+
+``categories_as_tabs:``
+-----------------------
+
+A list of the categories that are displayed as tabs in the user interface.
+Categories not listed here can still be searched with the :ref:`search-syntax`.
+
+.. code-block:: yaml
+
+  categories_as_tabs:
+    general:
+    images:
+    videos:
+    news:
+    map:
+    music:
+    it:
+    science:
+    files:
+    social media:
+
+Engines are added to ``categories:`` (compare :ref:`engine categories`), the
+categories listed in ``categories_as_tabs`` are shown as tabs in the UI.  If
+there are no active engines in a category, the tab is not displayed (e.g. if a
+user disables all engines in a category).
+
+On the preferences page (``/preferences``) -- under *engines* -- there is an
+additional tab, called *other*.  In this tab are all engines listed that are not
+in one of the UI tabs (not included in ``categories_as_tabs``).
+
+.. _settings engine:
+
+Engine settings
+===============
+
+.. sidebar:: Further reading ..
+
+   - :ref:`configured engines`
+   - :ref:`engines-dev`
+
+In the code example below a *full fledged* example of a YAML setup from a dummy
+engine is shown.  Most of the options have a default value or even are optional.
+
+.. code:: yaml
+
+   - name: example engine
+     engine: example
+     shortcut: demo
+     base_url: 'https://{language}.example.com/'
+     send_accept_language_header: false
+     categories: general
+     timeout: 3.0
+     api_key: 'apikey'
+     disabled: false
+     language: en_US
+     tokens: [ 'my-secret-token' ]
+     weight: 1
+     display_error_messages: true
+     about:
+        website: https://example.com
+        wikidata_id: Q306656
+        official_api_documentation: https://example.com/api-doc
+        use_official_api: true
+        require_api_key: true
+        results: HTML
+     enable_http: false
+     enable_http2: false
+     retries: 1
+     retry_on_http_error: true # or 403 or [404, 429]
+     max_connections: 100
+     max_keepalive_connections: 10
+     keepalive_expiry: 5.0
+     proxies:
+       http:
+         - http://proxy1:8080
+         - http://proxy2:8080
+       https:
+         - http://proxy1:8080
+         - http://proxy2:8080
+         - socks5://user:password@proxy3:1080
+         - socks5h://user:password@proxy4:1080
+
+``name`` :
+  Name that will be used across SearXNG to define this engine.  In settings, on
+  the result page...
+
+``engine`` :
+  Name of the python file used to handle requests and responses to and from this
+  search engine.
+
+``shortcut`` :
+  Code used to execute bang requests (in this case using ``!bi``)
+
+``base_url`` : optional
+  Part of the URL that should be stable across every request.  Can be useful to
+  use multiple sites using only one engine, or updating the site URL without
+  touching at the code.
+
+``send_accept_language_header`` :
+  Several engines that support languages (or regions) deal with the HTTP header
+  ``Accept-Language`` to build a response that fits to the locale.  When this
+  option is activated, the language (locale) that is selected by the user is used
+  to build and send a ``Accept-Language`` header in the request to the origin
+  search engine.
+
+.. _engine categories:
+
+``categories`` : optional
+  Specifies to which categories the engine should be added.  Engines can be
+  assigned to multiple categories.
+
+  Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the
+  UI.  A search in a tab (in the UI) will query all engines that are active in
+  this tab.  In the preferences page (``/preferences``) -- under *engines* --
+  users can select what engine should be active when querying in this tab.
+
+  Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines
+  in a category, regardless of whether they are active or not, or whether they
+  are in a tab of the UI or not.  For example, ``!dictionaries`` can be used to
+  query all search engines in that category (group).
+
+``timeout`` : optional
+  Timeout of the search with the current search engine.  **Be careful, it will
+  modify the global timeout of SearXNG.**
+
+``api_key`` : optional
+  In a few cases, using an API needs the use of a secret key.  How to obtain them
+  is described in the file.
+
+``disabled`` : optional
+  To disable by default the engine, but not deleting it.  It will allow the user
+  to manually activate it in the settings.
+
+``inactive``: optional
+  Remove the engine from the settings (*disabled & removed*).
+
+``language`` : optional
+  If you want to use another language for a specific engine, you can define it
+  by using the ISO code of language (and region), like ``fr``, ``en-US``,
+  ``de-DE``.
+
+``tokens`` : optional
+  A list of secret tokens to make this engine *private*, more details see
+  :ref:`private engines`.
+
+``weight`` : default ``1``
+  Weighting of the results of this engine.
+
+``display_error_messages`` : default ``true``
+  When an engine returns an error, the message is displayed on the user interface.
+
+``network`` : optional
+  Use the network configuration from another engine.
+  In addition, there are two default networks:
+
+  - ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
+  - ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
+
+.. note::
+
+   A few more options are possible, but they are pretty specific to some
+   engines, and so won't be described here.
+
+
+Example: Multilingual Search
+----------------------------
+
+SearXNG does not support true multilingual search.  You have to use the language
+prefix in your search query when searching in a different language.
+
+But there is a workaround: By adding a new search engine with a different
+language, SearXNG will search in your default and other language.
+
+Example configuration in settings.yml for a German and English speaker:
+
+.. code-block:: yaml
+
+    search:
+        default_lang : "de"
+        ...
+
+    engines:
+      - name : google english
+        engine : google
+        language : en
+        ...
+
+When searching, the default google engine will return German results and
+"google english" will return English results.
+
+
+.. _settings use_default_settings:
+
+use_default_settings
+====================
+
+.. sidebar:: ``use_default_settings: true``
+
+   - :ref:`settings location`
+   - :ref:`use_default_settings.yml`
+   - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
+
+The user defined ``settings.yml`` is loaded from the :ref:`settings location`
+and can relied on the default configuration :origin:`searx/settings.yml` using:
+
+ ``use_default_settings: true``
+
+``server:``
+  In the following example, the actual settings are the default settings defined
+  in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
+  the ``bind_address``:
+
+  .. code-block:: yaml
+
+    use_default_settings: true
+    server:
+        secret_key: "ultrasecretkey"   # change this!
+        bind_address: "0.0.0.0"
+
+``engines:``
+  With ``use_default_settings: true``, each settings can be override in a
+  similar way, the ``engines`` section is merged according to the engine
+  ``name``.  In this example, SearXNG will load all the default engines, will
+  enable the ``bing`` engine and define a :ref:`token <private engines>` for
+  the arch linux engine:
+
+  .. code-block:: yaml
+
+    use_default_settings: true
+    server:
+      secret_key: "ultrasecretkey"   # change this!
+    engines:
+      - name: arch linux wiki
+        tokens: ['$ecretValue']
+      - name: bing
+        disabled: false
+
+
+``engines:`` / ``remove:``
+  It is possible to remove some engines from the default settings. The following
+  example is similar to the above one, but SearXNG doesn't load the the google
+  engine:
+
+  .. code-block:: yaml
+
+    use_default_settings:
+      engines:
+        remove:
+          - google
+    server:
+      secret_key: "ultrasecretkey"   # change this!
+    engines:
+      - name: arch linux wiki
+        tokens: ['$ecretValue']
+
+``engines:`` / ``keep_only:``
+  As an alternative, it is possible to specify the engines to keep. In the
+  following example, SearXNG has only two engines:
+
+  .. code-block:: yaml
+
+    use_default_settings:
+      engines:
+        keep_only:
+          - google
+          - duckduckgo
+    server:
+      secret_key: "ultrasecretkey"   # change this!
+    engines:
+      - name: google
+        tokens: ['$ecretValue']
+      - name: duckduckgo
+        tokens: ['$ecretValue']

docs/admin/engines/sql-engines.rst

@@ -10,22 +10,12 @@ SQL Engines
    - `PostgreSQL <https://www.postgresql.org>`_
    - `MySQL <https://www.mysql.com>`_
 
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. sidebar:: info
-
-   Initial sponsored by `Search and Discovery Fund
-   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
-
 With the *SQL engines* you can bind SQL databases into SearXNG.  The following
 Relational Database Management System (RDBMS) are supported:
 
 - :ref:`engine sqlite`
 - :ref:`engine postgresql`
-- :ref:`engine mysql_server` & :ref:`engine mariadb_server`
+- :ref:`engine mysql_server`
 
 All of the engines above are just commented out in the :origin:`settings.yml
 <searx/settings.yml>`, as you have to set the required attributes for the
@@ -52,18 +42,6 @@ add them and limit the access by setting ``tokens`` as described in section
 :ref:`private engines`.
 
 
-Extra Dependencies
-==================
-
-For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
-install additional packages in Python's Virtual Environment of your SearXNG
-instance.  To switch into the environment (:ref:`searxng-src`) you can use
-:ref:`searxng.sh`::
-
-  $ sudo utils/searxng.sh instance cmd bash
-  (searxng-pyenv)$ pip install ...
-
-
 Configure the engines
 =====================
 
@@ -86,8 +64,45 @@ SQLite
 
    - :origin:`sqlite.py <searx/engines/sqlite.py>`
 
-.. automodule:: searx.engines.sqlite
-  :members:
+.. _MediathekView: https://mediathekview.de/
+
+SQLite is a small, fast and reliable SQL database engine.  It does not require
+any extra dependency.  To demonstrate the power of database engines, here is a
+more complex example which reads from a MediathekView_ (DE) movie database.  For
+this example of the SQlite engine download the database:
+
+- https://liste.mediathekview.de/filmliste-v2.db.bz2
+
+and unpack into ``searx/data/filmliste-v2.db``.  To search the database use e.g
+Query to test: ``!mediathekview concert``
+
+.. code:: yaml
+
+   - name: mediathekview
+     engine: sqlite
+     disabled: False
+     categories: general
+     result_template: default.html
+     database: searx/data/filmliste-v2.db
+     query_str:  >-
+       SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
+              COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
+              description AS content
+         FROM film
+        WHERE title LIKE :wildcard OR description LIKE :wildcard
+        ORDER BY duration DESC
+
+
+Extra Dependencies
+------------------
+
+For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
+install additional packages in Python's Virtual Environment of your SearXNG
+instance.  To switch into the environment (:ref:`searxng-src`) you can use
+:ref:`searxng.sh`::
+
+  $ sudo utils/searxng.sh instance cmd bash
+  (searxng-pyenv)$ pip install ...
 
 
 .. _engine postgresql:
@@ -100,10 +115,20 @@ PostgreSQL
 .. sidebar:: info
 
    - :origin:`postgresql.py <searx/engines/postgresql.py>`
-   - ``pip install`` `psycopg2-binary <psycopg2>`_
+   - ``pip install`` psycopg2_
 
-.. automodule:: searx.engines.postgresql
-  :members:
+PostgreSQL is a powerful and robust open source database.  Before configuring
+the PostgreSQL engine, you must install the dependency ``psychopg2``.  You can
+find an example configuration below:
+
+.. code:: yaml
+
+   - name: my_database
+     engine: postgresql
+     database: my_database
+     username: searxng
+     password: password
+     query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
 
 .. _engine mysql_server:
 
@@ -115,20 +140,27 @@ MySQL
    - :origin:`mysql_server.py <searx/engines/mysql_server.py>`
    - ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>`
 
+MySQL is said to be the most popular open source database. Before enabling MySQL
+engine, you must install the package ``mysql-connector-python``.
 
-.. automodule:: searx.engines.mysql_server
-  :members:
+The authentication plugin is configurable by setting ``auth_plugin`` in the
+attributes.  By default it is set to ``caching_sha2_password``.  This is an
+example configuration for querying a MySQL server:
 
-.. _engine mariadb_server:
+.. code:: yaml
 
-MariaDB
---------
-
-.. sidebar:: info
-
-   - :origin:`mariadb_server.py <searx/engines/mariadb_server.py>`
-   - ``pip install`` :pypi:`mariadb <mariadb>`
+   - name: my_database
+     engine: mysql_server
+     database: my_database
+     username: searxng
+     password: password
+     limit: 5
+     query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
 
 
-.. automodule:: searx.engines.mariadb_server
-  :members:
+Acknowledgment
+==============
+
+This development was sponsored by `Search and Discovery Fund
+<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
+

docs/admin/index.rst

@@ -4,8 +4,8 @@ Administrator documentation
 
 .. toctree::
    :maxdepth: 2
+   :caption: Contents
 
-   settings/index
    installation
    installation-docker
    installation-scripts
@@ -15,8 +15,7 @@ Administrator documentation
    installation-apache
    update-searxng
    answer-captcha
-   searx.favicons
-   searx.limiter
+   engines/index
    api
    architecture
    plugins

docs/admin/installation-apache.rst

@@ -61,7 +61,7 @@ section might give you some guidance.
    - `Apache Fedora`_
    - `Apache directives`_
 
-.. contents::
+.. contents:: Contents
    :depth: 2
    :local:
    :backlinks: entry
@@ -190,7 +190,7 @@ Debian:
 Apache modules
 --------------
 
-To load additional modules, in most distributions you have to uncomment the
+To load additional modules, in most distributions you have to un-comment the
 lines with the corresponding LoadModule_ directive, except in :ref:`Debian's
 Apache layout`.
 

docs/admin/installation-docker.rst

@@ -43,7 +43,7 @@ of this container:
 
 - enables :ref:`limiter <limiter>` to protect against bots
 - enables :ref:`image proxy <image_proxy>` for better privacy
-- enables :ref:`cache busting <static_use_hash>` to save bandwidth
+- enables :ref:`cache busting <static_use_hash>` to save bandwith
 
 ----
 
@@ -92,9 +92,6 @@ instance using `docker run <https://docs.docker.com/engine/reference/run/>`_:
                 searxng/searxng
    2f998.... # container's ID
 
-The environment variables UWSGI_WORKERS and UWSGI_THREADS overwrite the default
-number of UWSGI processes and UWSGI threads specified in `/etc/searxng/uwsgi.ini`.
-
 Open your WEB browser and visit the URL:
 
 .. code:: sh
@@ -111,7 +108,7 @@ can modify these files according to your needs and restart the Docker image.
 Use command ``container ls`` to list running containers, add flag `-a
 <https://docs.docker.com/engine/reference/commandline/container_ls>`__ to list
 exited containers also.  With ``container stop`` a running container can be
-stopped.  To get rid of a container use ``container rm``:
+stoped.  To get rid of a container use ``container rm``:
 
 .. code:: sh
 

docs/admin/installation-nginx.rst

@@ -41,7 +41,7 @@ section might give you some guidance.
    - `uWSGI support from nginx`_
 
 
-.. contents::
+.. contents:: Contents
    :depth: 2
    :local:
    :backlinks: entry

docs/admin/installation-searxng.rst

@@ -4,7 +4,7 @@
 Step by step installation
 =========================
 
-.. contents::
+.. contents:: Contents
    :depth: 2
    :local:
    :backlinks: entry
@@ -73,7 +73,7 @@ Configuration
 
 .. sidebar:: ``use_default_settings: True``
 
-   - :ref:`settings.yml`
+   - :ref:`settings global`
    - :ref:`settings location`
    - :ref:`settings use_default_settings`
    - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
@@ -86,7 +86,7 @@ below. This setup:
 
 - enables :ref:`limiter <limiter>` to protect against bots
 - enables :ref:`image proxy <image_proxy>` for better privacy
-- enables :ref:`cache busting <static_use_hash>` to save bandwidth
+- enables :ref:`cache busting <static_use_hash>` to save bandwith
 
 Modify the ``/etc/searxng/settings.yml`` to your needs:
 
@@ -96,7 +96,7 @@ Modify the ``/etc/searxng/settings.yml`` to your needs:
 
      .. literalinclude:: ../../utils/templates/etc/searxng/settings.yml
         :language: yaml
-        :end-before: # hostnames:
+        :end-before: # hostname_replace:
 
      To see the entire file jump to :origin:`utils/templates/etc/searxng/settings.yml`
 
@@ -104,7 +104,7 @@ Modify the ``/etc/searxng/settings.yml`` to your needs:
 
      .. literalinclude:: ../../searx/settings.yml
         :language: yaml
-        :end-before: # hostnames:
+        :end-before: # hostname_replace:
 
      To see the entire file jump to :origin:`searx/settings.yml`
 

docs/admin/installation-uwsgi.rst

@@ -9,7 +9,7 @@ uWSGI
    - `systemd.unit`_
    - `uWSGI Emperor`_
 
-.. contents::
+.. contents:: Contents
    :depth: 2
    :local:
    :backlinks: entry

docs/admin/searx.favicons.rst

@@ -1,251 +0,0 @@
-.. _favicons:
-
-========
-Favicons
-========
-
-.. sidebar:: warning
-
-   Don't activate the favicons before reading the documentation.
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-Activating the favicons in SearXNG is very easy, but this **generates a
-significantly higher load** in the client/server communication and increases
-resources needed on the server.
-
-To mitigate these disadvantages, various methods have been implemented,
-including a *cache*.  The cache must be parameterized according to your own
-requirements and maintained regularly.
-
-To activate favicons in SearXNG's result list, set a default
-``favicon_resolver`` in the :ref:`search <settings search>` settings:
-
-.. code:: yaml
-
-   search:
-     favicon_resolver: "duckduckgo"
-
-By default and without any extensions, SearXNG serves these resolvers:
-
-- ``duckduckgo``
-- ``allesedv``
-- ``google``
-- ``yandex``
-
-With the above setting favicons are displayed, the user has the option to
-deactivate this feature in his settings.  If the user is to have the option of
-selecting from several *resolvers*, a further setting is required / but this
-setting will be discussed :ref:`later <register resolvers>` in this article,
-first we have to setup the favicons cache.
-
-Infrastructure
-==============
-
-The infrastructure for providing the favicons essentially consists of three
-parts:
-
-- :py:obj:`Favicons-Proxy <.favicons.proxy>` (aka *proxy*)
-- :py:obj:`Favicons-Resolvers <.favicons.resolvers>` (aka *resolver*)
-- :py:obj:`Favicons-Cache <.favicons.cache>` (aka *cache*)
-
-To protect the privacy of users, the favicons are provided via a *proxy*.  This
-*proxy* is automatically activated with the above activation of a *resolver*.
-Additional requests are required to provide the favicons: firstly, the *proxy*
-must process the incoming requests and secondly, the *resolver* must make
-outgoing requests to obtain the favicons from external sources.
-
-A *cache* has been developed to massively reduce both, incoming and outgoing
-requests.  This *cache* is also activated automatically with the above
-activation of a *resolver*.  In its defaults, however, the *cache* is minimal
-and not well suitable for a production environment!
-
-.. _favicon cache setup:
-
-Setting up the cache
-====================
-
-To parameterize the *cache* and more settings of the favicons infrastructure, a
-TOML_ configuration is created in the file ``/etc/searxng/favicons.toml``.
-
-.. code:: toml
-
-   [favicons]
-
-   cfg_schema = 1   # config's schema version no.
-
-   [favicons.cache]
-
-   db_url = "/var/cache/searxng/faviconcache.db"  # default: "/tmp/faviconcache.db"
-   LIMIT_TOTAL_BYTES = 2147483648                 # 2 GB / default: 50 MB
-   # HOLD_TIME = 5184000                            # 60 days / default: 30 days
-   # BLOB_MAX_BYTES = 40960                         # 40 KB / default 20 KB
-   # MAINTENANCE_MODE = "off"                       # default: "auto"
-   # MAINTENANCE_PERIOD = 600                       # 10min / default: 1h
-
-:py:obj:`cfg_schema <.FaviconConfig.cfg_schema>`:
-  Is required to trigger any processes required for future upgrades / don't
-  change it.
-
-:py:obj:`cache.db_url <.FaviconCacheConfig.db_url>`:
-  The path to the (SQLite_) database file.  The default path is in the `/tmp`_
-  folder, which is deleted on every reboot and is therefore unsuitable for a
-  production environment.  The FHS_ provides the folder  for the
-  application cache
-
-  The FHS_ provides the folder `/var/cache`_ for the cache of applications, so a
-  suitable storage location of SearXNG's caches is folder ``/var/cache/searxng``.
-  In container systems, a volume should be mounted for this folder and in a
-  standard installation (compare :ref:`create searxng user`), the folder must be
-  created and the user under which the SearXNG process is running must be given
-  write permission to this folder.
-
-  .. code:: bash
-
-     $ sudo mkdir /var/cache/searxng
-     $ sudo chown root:searxng /var/cache/searxng/
-     $ sudo chmod g+w /var/cache/searxng/
-
-:py:obj:`cache.LIMIT_TOTAL_BYTES <.FaviconCacheConfig.LIMIT_TOTAL_BYTES>`:
-  Maximum of bytes stored in the cache of all blobs.  The limit is only reached
-  at each maintenance interval after which the oldest BLOBs are deleted; the
-  limit is exceeded during the maintenance period.
-
-  .. attention::
-
-     If the maintenance period is too long or maintenance is switched
-     off completely, the cache grows uncontrollably.
-
-SearXNG hosters can change other parameters of the cache as required:
-
-- :py:obj:`cache.HOLD_TIME <.FaviconCacheConfig.HOLD_TIME>`
-- :py:obj:`cache.BLOB_MAX_BYTES <.FaviconCacheConfig.BLOB_MAX_BYTES>`
-
-
-Maintenance of the cache
-------------------------
-
-Regular maintenance of the cache is required!  By default, regular maintenance
-is triggered automatically as part of the client requests:
-
-- :py:obj:`cache.MAINTENANCE_MODE <.FaviconCacheConfig.MAINTENANCE_MODE>` (default ``auto``)
-- :py:obj:`cache.MAINTENANCE_PERIOD <.FaviconCacheConfig.MAINTENANCE_PERIOD>` (default ``6000`` / 1h)
-
-As an alternative to maintenance as part of the client request process, it is
-also possible to carry out maintenance using an external process. For example,
-by creating a :man:`crontab` entry for maintenance:
-
-.. code:: bash
-
-   $ python -m searx.favicons cache maintenance
-
-The following command can be used to display the state of the cache:
-
-.. code:: bash
-
-   $ python -m searx.favicons cache state
-
-
-.. _favicon proxy setup:
-
-Proxy configuration
-===================
-
-Most of the options of the :py:obj:`Favicons-Proxy <.favicons.proxy>` are
-already set sensibly with settings from the :ref:`settings.yml <searxng
-settings.yml>` and should not normally be adjusted.
-
-.. code:: toml
-
-   [favicons.proxy]
-
-   max_age = 5184000             # 60 days / default: 7 days (604800 sec)
-
-
-:py:obj:`max_age <.FaviconProxyConfig.max_age>`:
-  The `HTTP Cache-Control max-age`_ response directive indicates that the
-  response remains fresh until N seconds after the response is generated.  This
-  setting therefore determines how long a favicon remains in the client's cache.
-  As a rule, in the favicons infrastructure of SearXNG's this setting only
-  affects favicons whose byte size exceeds :ref:`BLOB_MAX_BYTES <favicon cache
-  setup>` (the other favicons that are already in the cache are embedded as
-  `data URL`_ in the :py:obj:`generated HTML <.favicons.proxy.favicon_url>`,
-  which can greatly reduce the number of additional requests).
-
-.. _register resolvers:
-
-Register resolvers
-------------------
-
-A :py:obj:`resolver <.favicon.resolvers>` is a function that obtains the favicon
-from an external source.  The resolver functions available to the user are
-registered with their fully qualified name (FQN_) in a ``resolver_map``.
-
-If no ``resolver_map`` is defined in the ``favicon.toml``, the favicon
-infrastructure of SearXNG generates this ``resolver_map`` automatically
-depending on the ``settings.yml``.  SearXNG would automatically generate the
-following TOML configuration from the following YAML configuration:
-
-.. code:: yaml
-
-   search:
-     favicon_resolver: "duckduckgo"
-
-.. code:: toml
-
-   [favicons.proxy.resolver_map]
-
-   "duckduckgo" = "searx.favicons.resolvers.duckduckgo"
-
-If this automatism is not desired, then (and only then) a separate
-``resolver_map`` must be created.  For example, to give the user two resolvers to
-choose from, the following configuration could be used:
-
-.. code:: toml
-
-   [favicons.proxy.resolver_map]
-
-   "duckduckgo" = "searx.favicons.resolvers.duckduckgo"
-   "allesedv" = "searx.favicons.resolvers.allesedv"
-   # "google" = "searx.favicons.resolvers.google"
-   # "yandex" = "searx.favicons.resolvers.yandex"
-
-.. note::
-
-   With each resolver, the resource requirement increases significantly.
-
-The number of resolvers increases:
-
-- the number of incoming/outgoing requests and
-- the number of favicons to be stored in the cache.
-
-In the following we list the resolvers available in the core of SearXNG, but via
-the FQN_ it is also possible to implement your own resolvers and integrate them
-into the *proxy*:
-
-- :py:obj:`searx.favicons.resolvers.duckduckgo`
-- :py:obj:`searx.favicons.resolvers.allesedv`
-- :py:obj:`searx.favicons.resolvers.google`
-- :py:obj:`searx.favicons.resolvers.yandex`
-
-
-
-.. _SQLite:
-   https://www.sqlite.org/
-.. _FHS:
-   https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
-.. _`/var/cache`:
-   https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s05.html
-.. _`/tmp`:
-   https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html
-.. _TOML:
-    https://toml.io/en/
-.. _HTTP Cache-Control max-age:
-   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#response_directives
-.. _data URL:
-   https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs
-.. _FQN: https://en.wikipedia.org/wiki/Fully_qualified_name
-

docs/admin/searx.limiter.rst

@@ -1,17 +0,0 @@
-.. _limiter:
-
-=======
-Limiter
-=======
-
-.. sidebar:: info
-
-   The limiter requires a :ref:`Redis <settings redis>` database.
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.limiter
-   :members:

docs/admin/settings/index.rst

@@ -1,27 +0,0 @@
-.. _searxng settings.yml:
-
-========
-Settings
-========
-
-.. sidebar:: Further reading ..
-
-   - :ref:`engine settings`
-   - :ref:`engine file`
-
-.. toctree::
-   :maxdepth: 2
-
-   settings
-   settings_engine
-   settings_brand
-   settings_general
-   settings_search
-   settings_server
-   settings_ui
-   settings_redis
-   settings_outgoing
-   settings_categories_as_tabs
-
-
-

docs/admin/settings/settings.rst

@@ -1,117 +0,0 @@
-.. _settings.yml:
-
-================
-``settings.yml``
-================
-
-This page describe the options possibilities of the :origin:`searx/settings.yml`
-file.
-
-.. sidebar:: Further reading ..
-
-   - :ref:`use_default_settings.yml`
-   - :ref:`search API`
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. _settings location:
-
-settings.yml location
-=====================
-
-The initial ``settings.yml`` we be load from these locations:
-
-1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
-2. ``/etc/searxng/settings.yml``
-
-If these files don't exist (or are empty or can't be read), SearXNG uses the
-:origin:`searx/settings.yml` file.  Read :ref:`settings use_default_settings` to
-see how you can simplify your *user defined* ``settings.yml``.
-
-
-
-.. _settings use_default_settings:
-
-use_default_settings
-====================
-
-.. sidebar:: ``use_default_settings: true``
-
-   - :ref:`settings location`
-   - :ref:`use_default_settings.yml`
-   - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
-
-The user defined ``settings.yml`` is loaded from the :ref:`settings location`
-and can relied on the default configuration :origin:`searx/settings.yml` using:
-
- ``use_default_settings: true``
-
-``server:``
-  In the following example, the actual settings are the default settings defined
-  in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
-  the ``bind_address``:
-
-  .. code:: yaml
-
-    use_default_settings: true
-    server:
-        secret_key: "ultrasecretkey"   # change this!
-        bind_address: "0.0.0.0"
-
-``engines:``
-  With ``use_default_settings: true``, each settings can be override in a
-  similar way, the ``engines`` section is merged according to the engine
-  ``name``.  In this example, SearXNG will load all the default engines, will
-  enable the ``bing`` engine and define a :ref:`token <private engines>` for
-  the arch linux engine:
-
-  .. code:: yaml
-
-    use_default_settings: true
-    server:
-      secret_key: "ultrasecretkey"   # change this!
-    engines:
-      - name: arch linux wiki
-        tokens: ['$ecretValue']
-      - name: bing
-        disabled: false
-
-
-``engines:`` / ``remove:``
-  It is possible to remove some engines from the default settings. The following
-  example is similar to the above one, but SearXNG doesn't load the the google
-  engine:
-
-  .. code:: yaml
-
-    use_default_settings:
-      engines:
-        remove:
-          - google
-    server:
-      secret_key: "ultrasecretkey"   # change this!
-    engines:
-      - name: arch linux wiki
-        tokens: ['$ecretValue']
-
-``engines:`` / ``keep_only:``
-  As an alternative, it is possible to specify the engines to keep. In the
-  following example, SearXNG has only two engines:
-
-  .. code:: yaml
-
-    use_default_settings:
-      engines:
-        keep_only:
-          - google
-          - duckduckgo
-    server:
-      secret_key: "ultrasecretkey"   # change this!
-    engines:
-      - name: google
-        tokens: ['$ecretValue']
-      - name: duckduckgo
-        tokens: ['$ecretValue']

docs/admin/settings/settings_brand.rst

@@ -1,25 +0,0 @@
-.. _settings brand:
-
-==========
-``brand:``
-==========
-
-.. code:: yaml
-
-   brand:
-     issue_url: https://github.com/searxng/searxng/issues
-     docs_url: https://docs.searxng.org
-     public_instances: https://searx.space
-     wiki_url: https://github.com/searxng/searxng/wiki
-
-``issue_url`` :
-  If you host your own issue tracker change this URL.
-
-``docs_url`` :
-  If you host your own documentation change this URL.
-
-``public_instances`` :
-  If you host your own https://searx.space change this URL.
-
-``wiki_url`` :
-  Link to your wiki (or ``false``)

docs/admin/settings/settings_categories_as_tabs.rst

@@ -1,31 +0,0 @@
-.. _settings categories_as_tabs:
-
-=======================
-``categories_as_tabs:``
-=======================
-
-A list of the categories that are displayed as tabs in the user interface.
-Categories not listed here can still be searched with the :ref:`search-syntax`.
-
-.. code:: yaml
-
-  categories_as_tabs:
-    general:
-    images:
-    videos:
-    news:
-    map:
-    music:
-    it:
-    science:
-    files:
-    social media:
-
-Engines are added to ``categories:`` (compare :ref:`engine categories`), the
-categories listed in ``categories_as_tabs`` are shown as tabs in the UI.  If
-there are no active engines in a category, the tab is not displayed (e.g. if a
-user disables all engines in a category).
-
-On the preferences page (``/preferences``) -- under *engines* -- there is an
-additional tab, called *other*.  In this tab are all engines listed that are not
-in one of the UI tabs (not included in ``categories_as_tabs``).

docs/admin/settings/settings_engine.rst

@@ -1,244 +0,0 @@
-.. _settings engine:
-
-===========
-``engine:``
-===========
-
-.. sidebar:: Further reading ..
-
-   - :ref:`configured engines`
-   - :ref:`engines-dev`
-
-In the code example below a *full fledged* example of a YAML setup from a dummy
-engine is shown.  Most of the options have a default value or even are optional.
-
-.. hint::
-
-   A few more options are possible, but they are pretty specific to some
-   engines (:ref:`engine implementations`).
-
-.. code:: yaml
-
-   - name: example engine
-     engine: example
-     shortcut: demo
-     base_url: 'https://{language}.example.com/'
-     send_accept_language_header: false
-     categories: general
-     timeout: 3.0
-     api_key: 'apikey'
-     disabled: false
-     language: en_US
-     tokens: [ 'my-secret-token' ]
-     weight: 1
-     display_error_messages: true
-     about:
-        website: https://example.com
-        wikidata_id: Q306656
-        official_api_documentation: https://example.com/api-doc
-        use_official_api: true
-        require_api_key: true
-        results: HTML
-
-     # overwrite values from section 'outgoing:'
-     enable_http2: false
-     retries: 1
-     max_connections: 100
-     max_keepalive_connections: 10
-     keepalive_expiry: 5.0
-     using_tor_proxy: false
-     proxies:
-       http:
-         - http://proxy1:8080
-         - http://proxy2:8080
-       https:
-         - http://proxy1:8080
-         - http://proxy2:8080
-         - socks5://user:password@proxy3:1080
-         - socks5h://user:password@proxy4:1080
-
-     # other network settings
-     enable_http: false
-     retry_on_http_error: true # or 403 or [404, 429]
-
-
-``name`` :
-  Name that will be used across SearXNG to define this engine.  In settings, on
-  the result page...
-
-``engine`` :
-  Name of the python file used to handle requests and responses to and from this
-  search engine.
-
-``shortcut`` :
-  Code used to execute bang requests (in this case using ``!bi``)
-
-``base_url`` : optional
-  Part of the URL that should be stable across every request.  Can be useful to
-  use multiple sites using only one engine, or updating the site URL without
-  touching at the code.
-
-``send_accept_language_header`` :
-  Several engines that support languages (or regions) deal with the HTTP header
-  ``Accept-Language`` to build a response that fits to the locale.  When this
-  option is activated, the language (locale) that is selected by the user is used
-  to build and send a ``Accept-Language`` header in the request to the origin
-  search engine.
-
-.. _engine categories:
-
-``categories`` : optional
-  Specifies to which categories the engine should be added.  Engines can be
-  assigned to multiple categories.
-
-  Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the
-  UI.  A search in a tab (in the UI) will query all engines that are active in
-  this tab.  In the preferences page (``/preferences``) -- under *engines* --
-  users can select what engine should be active when querying in this tab.
-
-  Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines
-  in a category, regardless of whether they are active or not, or whether they
-  are in a tab of the UI or not.  For example, ``!dictionaries`` can be used to
-  query all search engines in that category (group).
-
-``timeout`` : optional
-  Timeout of the search with the current search engine.  Overwrites
-  ``request_timeout`` from :ref:`settings outgoing`.  **Be careful, it will
-  modify the global timeout of SearXNG.**
-
-``api_key`` : optional
-  In a few cases, using an API needs the use of a secret key.  How to obtain them
-  is described in the file.
-
-``disabled`` : optional
-  To disable by default the engine, but not deleting it.  It will allow the user
-  to manually activate it in the settings.
-
-``inactive``: optional
-  Remove the engine from the settings (*disabled & removed*).
-
-``language`` : optional
-  If you want to use another language for a specific engine, you can define it
-  by using the ISO code of language (and region), like ``fr``, ``en-US``,
-  ``de-DE``.
-
-``tokens`` : optional
-  A list of secret tokens to make this engine *private*, more details see
-  :ref:`private engines`.
-
-``weight`` : default ``1``
-  Weighting of the results of this engine.
-
-``display_error_messages`` : default ``true``
-  When an engine returns an error, the message is displayed on the user interface.
-
-``network`` : optional
-  Use the network configuration from another engine.
-  In addition, there are two default networks:
-
-  - ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
-  - ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
-
-``enable_http`` : optional
-  Enable HTTP for this engine (by default only HTTPS is enabled).
-
-``retry_on_http_error`` : optional
-  Retry request on some HTTP status code.
-
-  Example:
-
-  * ``true`` : on HTTP status code between 400 and 599.
-  * ``403`` : on HTTP status code 403.
-  * ``[403, 429]``: on HTTP status code 403 and 429.
-
-``proxies`` :
-  Overwrites proxy settings from :ref:`settings outgoing`.
-
-``using_tor_proxy`` :
-  Using tor proxy (``true``) or not (``false``) for this engine.  The default is
-  taken from ``using_tor_proxy`` of the :ref:`settings outgoing`.
-
-.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
-
-``max_keepalive_connection#s`` :
-  `Pool limit configuration`_, overwrites value ``pool_maxsize`` from
-   :ref:`settings outgoing` for this engine.
-
-``max_connections`` :
-  `Pool limit configuration`_, overwrites value ``pool_connections`` from
-  :ref:`settings outgoing` for this engine.
-
-``keepalive_expiry`` :
-  `Pool limit configuration`_, overwrites value ``keepalive_expiry`` from
-  :ref:`settings outgoing` for this engine.
-
-
-.. _private engines:
-
-Private Engines (``tokens``)
-============================
-
-Administrators might find themselves wanting to limit access to some of the
-enabled engines on their instances.  It might be because they do not want to
-expose some private information through :ref:`offline engines`.  Or they would
-rather share engines only with their trusted friends or colleagues.
-
-.. sidebar:: info
-
-   Initial sponsored by `Search and Discovery Fund
-   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
-
-To solve this issue the concept of *private engines* exists.
-
-A new option was added to engines named `tokens`.  It expects a list of strings.
-If the user making a request presents one of the tokens of an engine, they can
-access information about the engine and make search requests.
-
-Example configuration to restrict access to the Arch Linux Wiki engine:
-
-.. code:: yaml
-
-  - name: arch linux wiki
-    engine: archlinux
-    shortcut: al
-    tokens: [ 'my-secret-token' ]
-
-Unless a user has configured the right token, the engine is going to be hidden
-from them.  It is not going to be included in the list of engines on the
-Preferences page and in the output of `/config` REST API call.
-
-Tokens can be added to one's configuration on the Preferences page under "Engine
-tokens".  The input expects a comma separated list of strings.
-
-The distribution of the tokens from the administrator to the users is not carved
-in stone.  As providing access to such engines implies that the admin knows and
-trusts the user, we do not see necessary to come up with a strict process.
-Instead, we would like to add guidelines to the documentation of the feature.
-
-
-Example: Multilingual Search
-============================
-
-SearXNG does not support true multilingual search.  You have to use the language
-prefix in your search query when searching in a different language.
-
-But there is a workaround: By adding a new search engine with a different
-language, SearXNG will search in your default and other language.
-
-Example configuration in settings.yml for a German and English speaker:
-
-.. code-block:: yaml
-
-    search:
-        default_lang : "de"
-        ...
-
-    engines:
-      - name : google english
-        engine : google
-        language : en
-        ...
-
-When searching, the default google engine will return German results and
-"google english" will return English results.
-

docs/admin/settings/settings_general.rst

@@ -1,34 +0,0 @@
-.. _settings general:
-
-============
-``general:``
-============
-
-.. code:: yaml
-
-   general:
-     debug: false
-     instance_name:  "SearXNG"
-     privacypolicy_url: false
-     donation_url: false
-     contact_url: false
-     enable_metrics: true
-
-``debug`` : ``$SEARXNG_DEBUG``
-  Allow a more detailed log if you run SearXNG directly. Display *detailed* error
-  messages in the browser too, so this must be deactivated in production.
-
-``donation_url`` :
-  Set value to ``true`` to use your own donation page written in the
-  :ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable
-  the donation link altogether.
-
-``privacypolicy_url``:
-  Link to privacy policy.
-
-``contact_url``:
-  Contact ``mailto:`` address or WEB form.
-
-``enable_metrics``:
-  Enabled by default. Record various anonymous metrics available at ``/stats``,
-  ``/stats/errors`` and ``/preferences``.

docs/admin/settings/settings_outgoing.rst

@@ -1,110 +0,0 @@
-.. _settings outgoing:
-
-=============
-``outgoing:``
-=============
-
-Communication with search engines.
-
-.. code:: yaml
-
-   outgoing:
-     request_timeout: 2.0       # default timeout in seconds, can be override by engine
-     max_request_timeout: 10.0  # the maximum timeout in seconds
-     useragent_suffix: ""       # information like an email address to the administrator
-     pool_connections: 100      # Maximum number of allowable connections, or null
-                                # for no limits. The default is 100.
-     pool_maxsize: 10           # Number of allowable keep-alive connections, or null
-                                # to always allow. The default is 10.
-     enable_http2: true         # See https://www.python-httpx.org/http2/
-     # uncomment below section if you want to use a custom server certificate
-     # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
-     # and https://www.python-httpx.org/compatibility/#ssl-configuration
-     #  verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
-     #
-     # uncomment below section if you want to use a proxyq see: SOCKS proxies
-     #   https://2.python-requests.org/en/latest/user/advanced/#proxies
-     # are also supported: see
-     #   https://2.python-requests.org/en/latest/user/advanced/#socks
-     #
-     #  proxies:
-     #    all://:
-     #      - http://proxy1:8080
-     #      - http://proxy2:8080
-     #
-     #  using_tor_proxy: true
-     #
-     # Extra seconds to add in order to account for the time taken by the proxy
-     #
-     #  extra_proxy_timeout: 10.0
-     #
-
-``request_timeout`` :
-  Global timeout of the requests made to others engines in seconds.  A bigger
-  timeout will allow to wait for answers from slow engines, but in consequence
-  will slow SearXNG reactivity (the result page may take the time specified in the
-  timeout to load).  Can be override by ``timeout`` in the :ref:`settings engine`.
-
-``useragent_suffix`` :
-  Suffix to the user-agent SearXNG uses to send requests to others engines.  If an
-  engine wish to block you, a contact info here may be useful to avoid that.
-
-.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
-
-``pool_maxsize``:
-  Number of allowable keep-alive connections, or ``null`` to always allow.  The
-  default is 10.  See ``max_keepalive_connections`` `Pool limit configuration`_.
-
-``pool_connections`` :
-  Maximum number of allowable connections, or ``null`` # for no limits.  The
-  default is 100.  See ``max_connections`` `Pool limit configuration`_.
-
-``keepalive_expiry`` :
-  Number of seconds to keep a connection in the pool.  By default 5.0 seconds.
-  See ``keepalive_expiry`` `Pool limit configuration`_.
-
-.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
-
-``proxies`` :
-  Define one or more proxies you wish to use, see `httpx proxies`_.
-  If there are more than one proxy for one protocol (http, https),
-  requests to the engines are distributed in a round-robin fashion.
-
-``source_ips`` :
-  If you use multiple network interfaces, define from which IP the requests must
-  be made. Example:
-
-  * ``0.0.0.0`` any local IPv4 address.
-  * ``::`` any local IPv6 address.
-  * ``192.168.0.1``
-  * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
-  * ``fe80::60a2:1691:e5a2:ee1f``
-  * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
-  * ``[ 192.168.0.1, fe80::/126 ]``
-
-``retries`` :
-  Number of retry in case of an HTTP error.  On each retry, SearXNG uses an
-  different proxy and source ip.
-
-``enable_http2`` :
-  Enable by default. Set to ``false`` to disable HTTP/2.
-
-.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
-.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
-
-``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
-  Allow to specify a path to certificate.
-  see `httpx verification defaults`_.
-
-  In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
-  ``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
-  see `httpx ssl configuration`_.
-
-``max_redirects`` :
-  30 by default. Maximum redirect before it is an error.
-
-``using_tor_proxy`` :
-  Using tor proxy (``true``) or not (``false``) for all engines.  The default is
-  ``false`` and can be overwritten in the :ref:`settings engine`
-
-

docs/admin/settings/settings_redis.rst

@@ -1,49 +0,0 @@
-.. _settings redis:
-
-==========
-``redis:``
-==========
-
-.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
-
-A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
-will find a description to test your redis connection in SearXNG.  When using
-sockets, don't forget to check the access rights on the socket::
-
-  ls -la /usr/local/searxng-redis/run/redis.sock
-  srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
-
-In this example read/write access is given to the *searxng-redis* group.  To get
-access rights to redis instance (the socket), your SearXNG (or even your
-developer) account needs to be added to the *searxng-redis* group.
-
-``url`` : ``$SEARXNG_REDIS_URL``
-  URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
-
-    redis://[[username]:[password]]@localhost:6379/0
-    rediss://[[username]:[password]]@localhost:6379/0
-    unix://[[username]:[password]]@/path/to/socket.sock?db=0
-
-.. _Redis Developer Notes:
-
-Redis Developer Notes
-=====================
-
-To set up a local redis instance, first set the socket path of the Redis DB
-in your YAML setting:
-
-.. code:: yaml
-
-   redis:
-     url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
-
-Then use the following commands to install the redis instance (:ref:`manage
-redis.help`):
-
-.. code:: sh
-
-   $ ./manage redis.build
-   $ sudo -H ./manage redis.install
-   $ sudo -H ./manage redis.addgrp "${USER}"
-   # don't forget to logout & login to get member of group
-

docs/admin/settings/settings_search.rst

@@ -1,104 +0,0 @@
-.. _settings search:
-
-===========
-``search:``
-===========
-
-.. code:: yaml
-
-   search:
-     safe_search: 0
-     autocomplete: ""
-     favicon_resolver: ""
-     default_lang: ""
-     ban_time_on_fail: 5
-     max_ban_time_on_fail: 120
-     suspended_times:
-       SearxEngineAccessDenied: 86400
-       SearxEngineCaptcha: 86400
-       SearxEngineTooManyRequests: 3600
-       cf_SearxEngineCaptcha: 1296000
-       cf_SearxEngineAccessDenied: 86400
-       recaptcha_SearxEngineCaptcha: 604800
-     formats:
-       - html
-
-``safe_search``:
-  Filter results.
-
-  - ``0``: None
-  - ``1``: Moderate
-  - ``2``: Strict
-
-``autocomplete``:
-  Existing autocomplete backends, leave blank to turn it off.
-
-  - ``dbpedia``
-  - ``duckduckgo``
-  - ``google``
-  - ``mwmbl``
-  - ``startpage``
-  - ``swisscows``
-  - ``qwant``
-  - ``wikipedia``
-
-``favicon_resolver``:
-  To activate favicons in SearXNG's result list select a default
-  favicon-resolver, leave blank to turn off the feature.  Don't activate the
-  favicons before reading the :ref:`Favicons documentation <favicons>`.
-
-``default_lang``:
-  Default search language - leave blank to detect from browser information or
-  use codes from :origin:`searx/languages.py`.
-
-``languages``:
-  List of available languages - leave unset to use all codes from
-  :origin:`searx/languages.py`.  Otherwise list codes of available languages.
-  The ``all`` value is shown as the ``Default language`` in the user interface
-  (in most cases, it is meant to send the query without a language parameter ;
-  in some cases, it means the English language) Example:
-
-  .. code:: yaml
-
-     languages:
-       - all
-       - en
-       - en-US
-       - de
-       - it-IT
-       - fr
-       - fr-BE
-
-``ban_time_on_fail``:
-  Ban time in seconds after engine errors.
-
-``max_ban_time_on_fail``:
-  Max ban time in seconds after engine errors.
-
-``suspended_times``:
-  Engine suspension time after error (in seconds; set to 0 to disable)
-
-  ``SearxEngineAccessDenied``: 86400
-    For error "Access denied" and "HTTP error [402, 403]"
-
-  ``SearxEngineCaptcha``: 86400
-    For error "CAPTCHA"
-
-  ``SearxEngineTooManyRequests``: 3600
-    For error "Too many request" and "HTTP error 429"
-
-  Cloudflare CAPTCHA:
-     - ``cf_SearxEngineCaptcha``: 1296000
-     - ``cf_SearxEngineAccessDenied``: 86400
-
-  Google CAPTCHA:
-    - ``recaptcha_SearxEngineCaptcha``: 604800
-
-``formats``:
-  Result formats available from web, remove format to deny access (use lower
-  case).
-
-  - ``html``
-  - ``csv``
-  - ``json``
-  - ``rss``

docs/admin/settings/settings_server.rst

@@ -1,57 +0,0 @@
-.. _settings server:
-
-===========
-``server:``
-===========
-
-.. code:: yaml
-
-   server:
-       base_url: http://example.org/location  # change this!
-       port: 8888
-       bind_address: "127.0.0.1"
-       secret_key: "ultrasecretkey"           # change this!
-       limiter: false
-       public_instance: false
-       image_proxy: false
-       default_http_headers:
-         X-Content-Type-Options : nosniff
-         X-Download-Options : noopen
-         X-Robots-Tag : noindex, nofollow
-         Referrer-Policy : no-referrer
-
-``base_url`` : ``$SEARXNG_URL``
-  The base URL where SearXNG is deployed.  Used to create correct inbound links.
-
-``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS``
-  Port number and *bind address* of the SearXNG web application if you run it
-  directly using ``python searx/webapp.py``.  Doesn't apply to a SearXNG
-  services running behind a proxy and using socket communications.
-
-``secret_key`` : ``$SEARXNG_SECRET``
-  Used for cryptography purpose.
-
-``limiter`` :  ``$SEARXNG_LIMITER``
-  Rate limit the number of request on the instance, block some bots.  The
-  :ref:`limiter` requires a :ref:`settings redis` database.
-
-.. _public_instance:
-
-``public_instance`` :  ``$SEARXNG_PUBLIC_INSTANCE``
-
-  Setting that allows to enable features specifically for public instances (not
-  needed for local usage).  By set to ``true`` the following features are
-  activated:
-
-  - :py:obj:`searx.botdetection.link_token` in the :ref:`limiter`
-
-.. _image_proxy:
-
-``image_proxy`` : ``$SEARXNG_IMAGE_PROXY``
-  Allow your instance of SearXNG of being able to proxy images.  Uses memory space.
-
-.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
-
-``default_http_headers`` :
-  Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
-

docs/admin/settings/settings_ui.rst

@@ -1,70 +0,0 @@
-.. _settings ui:
-
-=======
-``ui:``
-=======
-
-.. _cache busting:
-   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
-
-.. code:: yaml
-
-   ui:
-     static_use_hash: false
-     default_locale: ""
-     query_in_title: false
-     infinite_scroll: false
-     center_alignment: false
-     cache_url: https://web.archive.org/web/
-     default_theme: simple
-     theme_args:
-       simple_style: auto
-     search_on_category_select: true
-     hotkeys: default
-
-.. _static_use_hash:
-
-``static_use_hash`` : ``$SEARXNG_STATIC_USE_HASH``
-  Enables `cache busting`_ of static files.
-
-``default_locale`` :
-  SearXNG interface language.  If blank, the locale is detected by using the
-  browser language.  If it doesn't work, or you are deploying a language
-  specific instance of searx, a locale can be defined using an ISO language
-  code, like ``fr``, ``en``, ``de``.
-
-``query_in_title`` :
-  When true, the result page's titles contains the query it decreases the
-  privacy, since the browser can records the page titles.
-
-``infinite_scroll``:
-  When true, automatically loads the next page when scrolling to bottom of the current page.
-
-``center_alignment`` : default ``false``
-  When enabled, the results are centered instead of being in the left (or RTL)
-  side of the screen.  This setting only affects the *desktop layout*
-  (:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
-
-.. cache_url:
-
-``cache_url`` : ``https://web.archive.org/web/``
-  URL prefix of the internet archive or cache, don't forget trailing slash (if
-  needed).  The default is https://web.archive.org/web/ alternatives are:
-
-  - https://webcache.googleusercontent.com/search?q=cache:
-  - https://archive.today/
-
-``default_theme`` :
-  Name of the theme you want to use by default on your SearXNG instance.
-
-``theme_args.simple_style``:
-  Style of simple theme: ``auto``, ``light``, ``dark``, ``black``
-
-``results_on_new_tab``:
-  Open result links in a new tab by default.
-
-``search_on_category_select``:
-  Perform search immediately if a category selected. Disable to select multiple categories.
-
-``hotkeys``:
-  Hotkeys to use in the search interface: ``default``, ``vim`` (Vim-like).

docs/admin/update-searxng.rst

@@ -9,7 +9,7 @@ SearXNG maintenance
    - :ref:`toolboxing`
    - :ref:`uWSGI maintenance`
 
-.. contents::
+.. contents:: Contents
    :depth: 2
    :local:
    :backlinks: entry
@@ -61,7 +61,7 @@ and then, to name just a few:
 - The image proxy morty is no longer needed, it has been replaced by the
   :ref:`image proxy <image_proxy>` from SearXNG.
 
-- To save bandwidth :ref:`cache busting <static_use_hash>` has been implemented.
+- To save bandwith :ref:`cache busting <static_use_hash>` has been implemented.
   To get in use, the ``static-expires`` needs to be set in the :ref:`uwsgi
   setup`.
 

docs/build-templates/filtron.rst

@@ -0,0 +1,53 @@
+.. START create user
+
+.. tabs::
+
+  .. group-tab:: bash
+
+    .. code-block:: sh
+
+      $ sudo -H useradd --shell /bin/bash --system \\
+          --home-dir \"$SERVICE_HOME\" \\
+          --comment \"Privacy-respecting metasearch engine\" $SERVICE_USER
+
+      $ sudo -H mkdir \"$SERVICE_HOME\"
+      $ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
+
+.. END create user
+
+.. START install go
+
+.. tabs::
+
+  .. group-tab:: os: linux / arch: amd64
+
+
+    .. code-block:: bash
+
+       $ cat > \"$GO_ENV\" <<EOF
+       export GOPATH=${SERVICE_HOME}/go-apps
+       export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin
+       EOF
+       $ sudo -i -u \"${SERVICE_USER}\"
+       (${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile
+       (${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local
+       (${SERVICE_USER}) $ wget --progress=bar -O \"${GO_VERSION}.linux-amd64.tar.gz\" \\
+                   \"${GO_DL_URL}/${GO_VERSION}.linux-amd64.tar.gz\"
+       (${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_VERSION}.linux-amd64.tar.gz\"
+       (${SERVICE_USER}) $ which go
+       ${SERVICE_HOME}/local/go/bin/go
+
+.. END install go
+
+.. START install filtron
+
+.. tabs::
+
+  .. group-tab:: bash
+
+    .. code-block:: bash
+
+       $ sudo -i -u \"${SERVICE_USER}\"
+       (${SERVICE_USER}) $ go get -v -u github.com/searxng/filtron
+
+.. END install filtron

docs/build-templates/morty.rst

@@ -0,0 +1,53 @@
+.. START create user
+
+.. tabs::
+
+  .. group-tab:: bash
+
+    .. code-block:: sh
+
+      $ sudo -H useradd --shell /bin/bash --system \\
+          --home-dir \"$SERVICE_HOME\" \\
+          --comment \"Privacy-respecting metasearch engine\" $SERVICE_USER
+
+      $ sudo -H mkdir \"$SERVICE_HOME\"
+      $ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
+
+.. END create user
+
+.. START install go
+
+.. tabs::
+
+  .. group-tab:: os: linux / arch: amd64
+
+
+    .. code-block:: bash
+
+       $ cat > \"$GO_ENV\" <<EOF
+       export GOPATH=${SERVICE_HOME}/go-apps
+       export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin
+       EOF
+       $ sudo -i -u \"${SERVICE_USER}\"
+       (${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile
+       (${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local
+       (${SERVICE_USER}) $ wget --progress=bar -O \"${GO_VERSION}.linux-amd64.tar.gz\" \\
+                   \"${GO_DL_URL}/${GO_VERSION}.linux-amd64.tar.gz\"
+       (${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_VERSION}.linux-amd64.tar.gz\"
+       (${SERVICE_USER}) $ which go
+       ${SERVICE_HOME}/local/go/bin/go
+
+.. END install go
+
+.. START install morty
+
+.. tabs::
+
+  .. group-tab:: bash
+
+    .. code-block:: bash
+
+       $ sudo -i -u \"${SERVICE_USER}\"
+       (${SERVICE_USER}) $ go get -v -u github.com/asciimoo/morty
+
+.. END install morty

docs/build-templates/searxng.rst

@@ -113,7 +113,7 @@ ${fedora_build}
 
        (${SERVICE_USER})$ command -v python && python --version
        $SEARXNG_PYENV/bin/python
-       Python 3.11.10
+       Python 3.8.1
 
        # update pip's boilerplate ..
        pip install -U pip
@@ -123,7 +123,7 @@ ${fedora_build}
 
        # jump to SearXNG's working tree and install SearXNG into virtualenv
        (${SERVICE_USER})$ cd \"$SEARXNG_SRC\"
-       (${SERVICE_USER})$ pip install --use-pep517 --no-build-isolation -e .
+       (${SERVICE_USER})$ pip install -e .
 
 
 .. END manage.sh update_packages

docs/conf.py

@@ -1,7 +1,7 @@
+# -*- coding: utf-8 -*-
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
 import  sys, os
-from pathlib import Path
 from pallets_sphinx_themes import ProjectLink
 
 from searx import get_setting
@@ -10,9 +10,10 @@ from searx.version import VERSION_STRING, GIT_URL, GIT_BRANCH
 # Project --------------------------------------------------------------
 
 project = 'SearXNG'
-copyright = 'SearXNG team'
-author = 'SearXNG team'
+copyright = '2021 SearXNG team, 2015-2021 Adam Tauber, Noémi Ványi'
+author = '2021 SearXNG team, 2015-2021 Adam Tauber'
 release, version = VERSION_STRING, VERSION_STRING
+
 SEARXNG_URL = get_setting('server.base_url') or 'https://example.org/searxng'
 ISSUE_URL = get_setting('brand.issue_url')
 DOCS_URL = get_setting('brand.docs_url')
@@ -21,9 +22,6 @@ PRIVACYPOLICY_URL = get_setting('general.privacypolicy_url')
 CONTACT_URL = get_setting('general.contact_url')
 WIKI_URL = get_setting('brand.wiki_url')
 
-SOURCEDIR = Path(__file__).parent.parent / "searx"
-os.environ['SOURCEDIR'] = str(SOURCEDIR)
-
 # hint: sphinx.ext.viewcode won't highlight when 'highlight_language' [1] is set
 #       to string 'none' [2]
 #
@@ -127,7 +125,6 @@ extensions = [
     "sphinx_tabs.tabs", # https://github.com/djungelorm/sphinx-tabs
     'myst_parser',  # https://www.sphinx-doc.org/en/master/usage/markdown.html
     'notfound.extension',  # https://github.com/readthedocs/sphinx-notfound-page
-    'sphinxcontrib.autodoc_pydantic',  # https://github.com/mansenfranzen/autodoc_pydantic
 ]
 
 autodoc_default_options = {
@@ -171,7 +168,6 @@ imgmath_image_format = 'svg'
 imgmath_font_size = 14
 # sphinx.ext.imgmath setup END
 
-html_show_sphinx = False
 html_theme_options = {"index_sidebar_logo": True}
 html_context = {"project_links": [] }
 html_context["project_links"].append(ProjectLink("Source", GIT_URL + '/tree/' + GIT_BRANCH))

docs/dev/contribution_guide.rst

@@ -4,7 +4,7 @@
 How to contribute
 =================
 
-.. contents::
+.. contents:: Contents
    :depth: 2
    :local:
    :backlinks: entry
@@ -148,7 +148,7 @@ live build
 Live build is like WYSIWYG.  If you want to edit the documentation, its
 recommended to use.  The Makefile target ``docs.live`` builds the docs, opens
 URL in your favorite browser and rebuilds every time a reST file has been
-changed (:ref:`make docs.clean`).
+changed.
 
 .. code:: sh
 
@@ -183,8 +183,3 @@ commit and push:
 .. code:: sh
 
    $ make docs.clean docs.gh-pages
-
-.. attention::
-
-   If you are working in your own brand, don't forget to adjust your
-   :ref:`settings brand`.

docs/dev/engine_overview.rst

@@ -4,11 +4,6 @@
 Engine Overview
 ===============
 
-.. contents::
-   :depth: 3
-   :local:
-   :backlinks: entry
-
 .. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
 
 .. sidebar:: Further reading ..
@@ -16,6 +11,10 @@ Engine Overview
    - :ref:`configured engines`
    - :ref:`settings engine`
 
+.. contents::
+   :depth: 3
+   :backlinks: entry
+
 SearXNG is a metasearch-engine_, so it uses different search engines to provide
 better results.
 
@@ -50,12 +49,12 @@ Engine File
    categories              list        categories, in which the engine is working
    paging                  boolean     support multiple pages
    time_range_support      boolean     support search time range
-   engine_type             str         - ``online`` :ref:`[ref] <online engines>` by
+   engine_type             str         - ``online`` :ref:`[ref] <demo online engine>` by
                                          default, other possibles values are:
                                        - ``offline`` :ref:`[ref] <offline engines>`
-                                       - ``online_dictionary`` :ref:`[ref] <online dictionary>`
-                                       - ``online_currency`` :ref:`[ref] <online currency>`
-                                       - ``online_url_search`` :ref:`[ref] <online url search>`
+                                       - ``online_dictionary``
+                                       - ``online_currency``
+                                       - ``online_url_search``
    ======================= =========== ========================================================
 
 .. _engine settings:
@@ -87,8 +86,8 @@ For a more  detailed description, see :ref:`settings engine` in the :ref:`settin
 Overrides
 ---------
 
-A few of the options have default values in the namespace of the engine's python
-module, but are often overwritten by the settings.  If ``None`` is assigned to an
+A few of the options have default values in the namespace of engine's python
+modul, but are often overwritten by the settings.  If ``None`` is assigned to an
 option in the engine file, it has to be redefined in the settings, otherwise
 SearXNG will not start with that engine (global names with a leading underline can
 be ``None``).
@@ -240,18 +239,12 @@ following parameters can be used to specify a search request:
 .. _engine results:
 .. _engine media types:
 
-Result Types (``template``)
-===========================
+Media Types
+===========
 
 Each result item of an engine can be of different media-types.  Currently the
-following media-types are supported.  To set another media-type as
-:ref:`template default`, the parameter ``template`` must be set to the desired
-type.
-
-.. _template default:
-
-``default``
------------
+following media-types are supported.  To set another media-type as ``default``,
+the parameter ``template`` must be set to the desired type.
 
 .. table::  Parameter of the **default** media type:
    :width: 100%
@@ -266,65 +259,23 @@ type.
    ========================= =====================================================
 
 
-.. _template images:
-
-``images``
-----------
-
-.. list-table:: Parameter of the **images** media type
-   :header-rows: 2
+.. table::  Parameter of the **images** media type:
    :width: 100%
 
-   * - result-parameter
-     - Python type
-     - information
+   ========================= =====================================================
+   result-parameter          information
+   ------------------------- -----------------------------------------------------
+   template                  is set to ``images.html``
+   ========================= =====================================================
+   url                       string, url to the result site
+   title                     string, title of the result *(partly implemented)*
+   content                   *(partly implemented)*
+   publishedDate             :py:class:`datetime.datetime`,
+                             time of publish *(partly implemented)*
+   img\_src                  string, url to the result image
+   thumbnail\_src            string, url to a small-preview image
+   ========================= =====================================================
 
-   * - template
-     - :py:class:`str`
-     - is set to ``images.html``
-
-   * - url
-     - :py:class:`str`
-     - url to the result site
-
-   * - title
-     - :py:class:`str`
-     - title of the result
-
-   * - content
-     - :py:class:`str`
-     - description of the image
-
-   * - publishedDate
-     - :py:class:`datetime <datetime.datetime>`
-     - time of publish
-
-   * - img_src
-     - :py:class:`str`
-     - url to the result image
-
-   * - thumbnail_src
-     - :py:class:`str`
-     - url to a small-preview image
-
-   * - resolution
-     - :py:class:`str`
-     - the resolution of the image (e.g. ``1920 x 1080`` pixel)
-
-   * - img_format
-     - :py:class:`str`
-     - the format of the image (e.g. ``png``)
-
-   * - filesize
-     - :py:class:`str`
-     - size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation
-       (e.g. ``MB`` for 1024 \* 1024 Bytes filesize).
-
-
-.. _template videos:
-
-``videos``
-----------
 
 .. table::  Parameter of the **videos** media type:
    :width: 100%
@@ -339,16 +290,8 @@ type.
    content                   *(not implemented yet)*
    publishedDate             :py:class:`datetime.datetime`, time of publish
    thumbnail                 string, url to a small-preview image
-   length                    :py:class:`datetime.timedelta`, duration of result
-   views                     string, view count in humanized number format
    ========================= =====================================================
 
-
-.. _template torrent:
-
-``torrent``
------------
-
 .. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
 
 .. table::  Parameter of the **torrent** media type:
@@ -372,12 +315,6 @@ type.
    torrentfile               string, torrentfile of the result
    ========================= =====================================================
 
-
-.. _template map:
-
-``map``
--------
-
 .. table::  Parameter of the **map** media type:
    :width: 100%
 
@@ -405,12 +342,6 @@ type.
    address.country           country of object
    ========================= =====================================================
 
-
-.. _template paper:
-
-``paper``
----------
-
 .. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
 .. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
 
@@ -499,73 +430,3 @@ type.
    * - html_url
      - :py:class:`str`
      - URL to full article, HTML version
-
-
-.. _template packages:
-
-``packages``
-------------
-
-.. list-table:: Parameter of the **packages** media type
-   :header-rows: 2
-   :width: 100%
-
-   * - result-parameter
-     - Python type
-     - information
-
-   * - template
-     - :py:class:`str`
-     - is set to ``packages.html``
-
-   * - title
-     - :py:class:`str`
-     - title of the result
-
-   * - content
-     - :py:class:`str`
-     - abstract
-
-   * - package_name
-     - :py:class:`str`
-     - the name of the package
-
-   * - version
-     - :py:class:`str`
-     - the current version of the package
-
-   * - maintainer
-     - :py:class:`str`
-     - the maintainer or author of the project
-
-   * - publishedDate
-     - :py:class:`datetime <datetime.datetime>`
-     - date of latest update or release
-
-   * - tags
-     - :py:class:`List <list>`\ [\ :py:class:`str`\ ]
-     - free tag list
-
-   * - popularity
-     - :py:class:`str`
-     - the popularity of the package, e.g. rating or download count
-
-   * - license_name
-     - :py:class:`str`
-     - the name of the license
-
-   * - license_url
-     - :py:class:`str`
-     - the web location of a license copy
-
-   * - homepage
-     - :py:class:`str`
-     - the url of the project's homepage
-
-   * - source_code_url
-     - :py:class:`str`
-     - the location of the project's source code
-
-   * - links
-     - :py:class:`dict`
-     - additional links in the form of ``{'link_name': 'http://example.com'}``

docs/dev/engines/engines.rst

@@ -1,9 +0,0 @@
-.. _searx.engines loader:
-
-========================
-SearXNG's engines loader
-========================
-
-.. automodule:: searx.engines
-  :members:
-

docs/dev/engines/index.rst

@@ -1,107 +0,0 @@
-.. _engine implementations:
-
-======================
-Engine Implementations
-======================
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-
-.. toctree::
-   :caption: Framework Components
-   :maxdepth: 2
-
-   enginelib
-   engines
-   engine_overview
-
-
-Engine Types
-============
-
-The :py:obj:`engine_type <searx.enginelib.Engine.engine_type>` of an engine
-determines which :ref:`search processor <searx.search.processors>` is used by
-the engine.
-
-In this section a list of the engines that are documented is given, a complete
-list of the engines can be found in the source under: :origin:`searx/engines`.
-
-.. _online engines:
-
-Online Engines
---------------
-
-.. sidebar:: info
-
-   - :py:obj:`processors.online <searx.search.processors.online>`
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   demo/demo_online
-   xpath
-   mediawiki
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   online/*
-
-.. _offline engines:
-
-Offline Engines
----------------
-
-.. sidebar:: info
-
-   - :py:obj:`processors.offline <searx.search.processors.offline>`
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   offline_concept
-   demo/demo_offline
-   offline/*
-
-.. _online url search:
-
-Online URL Search
------------------
-
-.. sidebar:: info
-
-   - :py:obj:`processors.online_url_search <searx.search.processors.online_url_search>`
-
-.. toctree::
-   :maxdepth: 1
-   :glob:
-
-   online_url_search/*
-
-.. _online currency:
-
-Online Currency
----------------
-
-.. sidebar:: info
-
-   - :py:obj:`processors.online_currency <searx.search.processors.online_currency>`
-
-*no engine of this type is documented yet / comming soon*
-
-.. _online dictionary:
-
-Online Dictionary
------------------
-
-.. sidebar:: info
-
-   - :py:obj:`processors.online_dictionary <searx.search.processors.online_dictionary>`
-
-*no engine of this type is documented yet / comming soon*

docs/dev/engines/mediawiki.rst

@@ -1,13 +0,0 @@
-.. _mediawiki engine:
-
-================
-MediaWiki Engine
-================
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.mediawiki
-  :members:

docs/dev/engines/offline/command-line-engines.rst

@@ -1,23 +0,0 @@
-.. _engine command:
-
-====================
-Command Line Engines
-====================
-
-.. sidebar:: info
-
-   - :origin:`command.py <searx/engines/command.py>`
-   - :ref:`offline engines`
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. sidebar:: info
-
-   Initial sponsored by `Search and Discovery Fund
-   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
-
-.. automodule:: searx.engines.command
-  :members:

docs/dev/engines/offline/search-indexer-engines.rst

@@ -1,62 +0,0 @@
-=================
-Local Search APIs
-=================
-
-.. sidebar:: further read
-
-   - `Comparison to alternatives
-     <https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
-
-.. contents::
-   :depth: 1
-   :local:
-   :backlinks: entry
-
-.. sidebar:: info
-
-   Initial sponsored by `Search and Discovery Fund
-   <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
-
-Administrators might find themselves wanting to integrate locally running search
-engines.  The following ones are supported for now:
-
-* `Elasticsearch`_
-* `Meilisearch`_
-* `Solr`_
-
-Each search engine is powerful, capable of full-text search.  All of the engines
-above are added to ``settings.yml`` just commented out, as you have to
-``base_url`` for all them.
-
-Please note that if you are not using HTTPS to access these engines, you have to
-enable HTTP requests by setting ``enable_http`` to ``True``.
-
-Furthermore, if you do not want to expose these engines on a public instance,
-you can still add them and limit the access by setting ``tokens`` as described
-in section :ref:`private engines`.
-
-.. _engine meilisearch:
-
-MeiliSearch
-===========
-
-.. automodule:: searx.engines.meilisearch
-  :members:
-
-
-.. _engine elasticsearch:
-
-Elasticsearch
-=============
-
-.. automodule:: searx.engines.elasticsearch
-  :members:
-
-.. _engine solr:
-
-Solr
-====
-
-.. automodule:: searx.engines.solr
-  :members:
-

docs/dev/engines/online/alpinelinux.rst

@@ -1,13 +0,0 @@
-.. _alpinelinux engine:
-
-=====================
-Alpine Linux Packages
-=====================
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.alpinelinux
-   :members:

docs/dev/engines/online/annas_archive.rst

@@ -1,13 +0,0 @@
-.. _annas_archive engine:
-
-==============
-Anna's Archive
-==============
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.annas_archive
-   :members:

docs/dev/engines/online/bpb.rst

@@ -1,13 +0,0 @@
-.. _bpb engine:
-
-===
-Bpb
-===
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.bpb
-  :members:

docs/dev/engines/online/brave.rst

@@ -1,13 +0,0 @@
-.. _brave engine:
-
-=============
-Brave Engines
-=============
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.brave
-  :members:

docs/dev/engines/online/bt4g.rst

@@ -1,14 +0,0 @@
-.. _bt4g engine:
-
-====
-BT4G
-====
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.bt4g
-  :members:
-

docs/dev/engines/online/discourse.rst

@@ -1,8 +0,0 @@
-.. _discourse engine:
-
-================
-Discourse Forums
-================
-
-.. automodule:: searx.engines.discourse
-   :members:

docs/dev/engines/online/geizhals.rst

@@ -1,8 +0,0 @@
-.. _gitea geizhals:
-
-========
-Geizhals
-========
-
-.. automodule:: searx.engines.geizhals
-  :members:

docs/dev/engines/online/gitea.rst

@@ -1,8 +0,0 @@
-.. _gitea engine:
-
-=====
-Gitea
-=====
-
-.. automodule:: searx.engines.gitea
-  :members:

docs/dev/engines/online/gitlab.rst

@@ -1,8 +0,0 @@
-.. _gitlab engine:
-
-======
-GitLab
-======
-
-.. automodule:: searx.engines.gitlab
-   :members:

docs/dev/engines/online/lemmy.rst

@@ -1,13 +0,0 @@
-.. _lemmy engine:
-
-=====
-Lemmy
-=====
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.lemmy
-   :members:

docs/dev/engines/online/loc.rst

@@ -1,13 +0,0 @@
-.. _loc engine:
-
-===================
-Library of Congress
-===================
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.loc
-  :members:

docs/dev/engines/online/mastodon.rst

@@ -1,13 +0,0 @@
-.. _mastodon engine:
-
-========
-Mastodon
-========
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.mastodon
-  :members:

docs/dev/engines/online/moviepilot.rst

@@ -1,13 +0,0 @@
-.. _moviepilot engine:
-
-==========
-Moviepilot
-==========
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.moviepilot
-   :members:

docs/dev/engines/online/mrs.rst

@@ -1,13 +0,0 @@
-.. _mrs engine:
-
-=========================
-Matrix Rooms Search (MRS)
-=========================
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.mrs
-   :members:

docs/dev/engines/online/mullvad_leta.rst

@@ -1,13 +0,0 @@
-.. _voidlinux mullvad_leta:
-
-============
-Mullvad-Leta
-============
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.mullvad_leta
-   :members:

docs/dev/engines/online/mwmbl.rst

@@ -1,27 +0,0 @@
-.. _Mwmbl engine:
-
-============
-Mwmbl Engine
-============
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-
-.. _mwmbl web engine:
-
-Mwmbl WEB
-=========
-
-.. automodule:: searx.engines.mwmbl
-  :members:
-
-
-.. _mwmbl autocomplete:
-
-Mwmbl Autocomplete
-==================
-
-.. autofunction:: searx.autocomplete.mwmbl

docs/dev/engines/online/odysee.rst

@@ -1,13 +0,0 @@
-.. _odysee engine:
-
-======
-Odysee
-======
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.odysee
-  :members:

docs/dev/engines/online/piped.rst

@@ -1,13 +0,0 @@
-.. _piped engine:
-
-=====
-Piped
-=====
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.piped
-  :members:

docs/dev/engines/online/presearch.rst

@@ -1,13 +0,0 @@
-.. _engine presearch:
-
-================
-Presearch Engine
-================
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.presearch
-   :members:

docs/dev/engines/online/qwant.rst

@@ -1,13 +0,0 @@
-.. _qwant engine:
-
-=====
-Qwant
-=====
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.qwant
-   :members:

docs/dev/engines/online/radio_browser.rst

@@ -1,13 +0,0 @@
-.. _RadioBrowser engine:
-
-============
-RadioBrowser
-============
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.radio_browser
-   :members:

docs/dev/engines/online/recoll.rst

@@ -1,13 +0,0 @@
-.. _engine recoll:
-
-=============
-Recoll Engine
-=============
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.recoll
-   :members:

docs/dev/engines/online/seekr.rst

@@ -1,13 +0,0 @@
-.. _seekr engine:
-
-=============
-Seekr Engines
-=============
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.seekr
-  :members:

docs/dev/engines/online/tagesschau.rst

@@ -1,13 +0,0 @@
-.. _tagesschau engine:
-
-==============
-Tagesschau API
-==============
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.tagesschau
-  :members:

docs/dev/engines/online/torznab.rst

@@ -1,13 +0,0 @@
-.. _torznab engine:
-
-==============
-Torznab WebAPI
-==============
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.torznab
-   :members:

docs/dev/engines/online/void.rst

@@ -1,13 +0,0 @@
-.. _voidlinux engine:
-
-==========================
-Void Linux binary packages
-==========================
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.voidlinux
-  :members:

docs/dev/engines/online/wallhaven.rst

@@ -1,13 +0,0 @@
-.. _wallhaven engine:
-
-=========
-Wallhaven
-=========
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.wallhaven
-  :members:

docs/dev/engines/online/yacy.rst

@@ -1,13 +0,0 @@
-.. _yacy engine:
-
-====
-Yacy
-====
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.yacy
-  :members:

docs/dev/engines/online/zlibrary.rst

@@ -1,13 +0,0 @@
-.. _zlibrary engine:
-
-=========
-Z-Library
-=========
-
-.. contents:: Contents
-   :depth: 2
-   :local:
-   :backlinks: entry
-
-.. automodule:: searx.engines.zlibrary
-  :members:

docs/dev/index.rst

@@ -4,11 +4,12 @@ Developer documentation
 
 .. toctree::
    :maxdepth: 2
+   :caption: Contents
 
    quickstart
-   rtm_asdf
    contribution_guide
-   engines/index
+   engine_overview
+   offline_engines
    search_api
    plugins
    translation

docs/dev/lxcdev.rst

@@ -22,7 +22,7 @@ In this article we will show, how you can make use of Linux Containers (LXC_) in
    section :ref:`internet connectivity docker`.
 
 
-.. contents::
+.. contents:: Contents
    :depth: 2
    :local:
    :backlinks: entry
@@ -148,7 +148,7 @@ services quite easy.
 In the example above the SearXNG instance in the container is wrapped to
 ``http://n.n.n.140/searxng`` to the HOST system.  Note, on your HOST system, the
 IP of your ``searxng-archlinux`` container is different to this example.  To
-test the instance in the container from outside of the container, in your WEB
+test the instance in the conatiner from outside of the container, in your WEB
 browser on your desktop just open the URL reported in your installation
 
 .. _working in containers:
@@ -255,7 +255,7 @@ With the use of the :ref:`searxng.sh` the SearXNG service was installed as
 With the command above, we stopped the SearXNG uWSGI-App in the archlinux
 container.
 
-The uWSGI-App for the archlinux distros is configured in
+The uWSGI-App for the archlinux dsitros is configured in
 :origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at
 least you should attend the settings of ``uid``, ``chdir``, ``env`` and
 ``http``::
@@ -272,7 +272,7 @@ shares the root folder of the repository and the command ``utils/lxc.sh cmd``
 handles relative path names **transparent**.
 
 To wrap the SearXNG installation in the container into a developer one, we
-simple have to create a symlink to the **transparent** repository from the
+simple have to create a smylink to the **transparent** reposetory from the
 desktop.  Now lets replace the repository at ``searxng-src`` in the container
 with the working tree from outside of the container:
 

docs/dev/makefile.rst

@@ -1,48 +1,31 @@
 .. _makefile:
 
-=======================
-Makefile & ``./manage``
-=======================
+========
+Makefile
+========
 
 .. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction
 
-All relevant build and development tasks are implemented in the
-:origin:`./manage <manage>` script and for CI or IDE integration a small
-:origin:`Makefile` wrapper is available.  If you are not familiar with
-Makefiles, we recommend to read gnu-make_ introduction.
-
 .. sidebar:: build environment
 
    Before looking deeper at the targets, first read about :ref:`make
    install`.
 
-   To install developer requirements follow :ref:`buildhosts`.
+   To install system requirements follow :ref:`buildhosts`.
 
-
-.. contents::
-   :depth: 2
-   :local:
-   :backlinks: entry
+All relevant build tasks are implemented in :origin:`manage` and for CI or
+IDE integration a small ``Makefile`` wrapper is available.  If you are not
+familiar with Makefiles, we recommend to read gnu-make_ introduction.
 
 The usage is simple, just type ``make {target-name}`` to *build* a target.
 Calling the ``help`` target gives a first overview (``make help``):
 
-.. tabs::
+.. program-output:: bash -c "cd ..; make --no-print-directory help"
 
-  .. group-tab:: ``make``
-
-     .. program-output:: bash -c "cd ..; make --no-print-directory help"
-
-
-  .. group-tab:: ``./manage``
-
-     The Makefile targets are implemented for comfort, if you can do without
-     tab-completion and need to have a more granular control, use
-     :origin:`manage` without the Makefile wrappers.
-
-     .. code:: sh
-
-        $ ./manage help
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
 
 .. _make install:
 
@@ -61,9 +44,13 @@ working tree and release a ``make install`` to get a virtualenv with a
    $ make install
    PYENV     [virtualenv] installing ./requirements*.txt into local/py3
    ...
-   PYENV     [install] pip install --use-pep517 --no-build-isolation -e 'searx[test]'
+   PYENV     OK
+   PYENV     [install] pip install -e 'searx[test]'
    ...
-   Successfully installed searxng-2023.7.19+a446dea1b
+   Successfully installed argparse-1.4.0 searx
+   BUILDENV  INFO:searx:load the default settings from ./searx/settings.yml
+   BUILDENV  INFO:searx:Initialisation done
+   BUILDENV  build utils/brand.env
 
 If you release ``make install`` multiple times the installation will only
 rebuild if the sha256 sum of the *requirement files* fails.  With other words:
@@ -78,9 +65,13 @@ the check fails if you edit the requirements listed in
    ...
    PYENV     [virtualenv] installing ./requirements*.txt into local/py3
    ...
-   PYENV     [install] pip install --use-pep517 --no-build-isolation -e 'searx[test]'
+   PYENV     OK
+   PYENV     [install] pip install -e 'searx[test]'
    ...
-   Successfully installed searxng-2023.7.19+a446dea1b
+   Successfully installed argparse-1.4.0 searx
+   BUILDENV  INFO:searx:load the default settings from ./searx/settings.yml
+   BUILDENV  INFO:searx:Initialisation done
+   BUILDENV  build utils/brand.env
 
 .. sidebar:: drop environment
 
@@ -90,6 +81,67 @@ the check fails if you edit the requirements listed in
 If you think, something goes wrong with your ./local environment or you change
 the :origin:`setup.py` file, you have to call :ref:`make clean`.
 
+.. _make buildenv:
+
+``make buildenv``
+=================
+
+Rebuild instance's environment with the modified settings from the
+:ref:`settings brand` and :ref:`settings server` section of your
+:ref:`settings.yml <settings location>`.
+
+  What is the :origin:`utils/brand.env` needed for and why do you need to rebuild
+  it if necessary?
+
+  Short answer: :ref:`installation and maintenance <searxng maintenance>`
+  scripts are running outside of instance's runtime environment and need some
+  values defined in the runtime environment.
+
+All the SearXNG setups are centralized in the :ref:`settings.yml` file.  This
+setup is available as long we are in a *installed instance*.  E.g. the
+*installed instance* on the server or the *installed developer instance* at
+``./local`` (the later one is created by a :ref:`make install <make install>` or
+:ref:`make run <make run>`).
+
+Tasks running outside of an *installed instance*, especially :ref:`installation
+and maintenance <searxng maintenance>` tasks running at (pre-) installation time
+do not have access to the SearXNG setup (from a *installed instance*).  Those
+tasks need a *build environment*.
+
+The ``make buildenv`` target will update the *build environment* in:
+
+- :origin:`utils/brand.env`
+
+Tasks running outside of an *installed instance*, need the following settings
+from the YAML configuration:
+
+- ``SEARXNG_URL`` from :ref:`server.base_url <settings  server>` (aka
+  ``PUBLIC_URL``)
+- ``SEARXNG_BIND_ADDRESS`` from :ref:`server.bind_address <settings server>`
+- ``SEARXNG_PORT`` from :ref:`server.port <settings server>`
+
+The ``GIT_URL`` and ``GIT_BRANCH`` in the origin:`utils/brand.env` file, are
+readed from the git VCS and the branch that is checked out when ``make
+buildenv`` command runs.
+
+.. _brand:
+
+**I would like to create my own brand, how should I proceed?**
+
+Create a remote branch (``example.org``), checkout the remote branch (on your
+local developer desktop) and in the :origin:`searx/settings.yml` file in the
+:ref:`settings server` section set ``base_url``.  Run ``make buildenv`` and
+create a commit for your brand.
+
+On your server you clone the branch (``example.org``) into your HOME folder
+``~`` from where you run the :ref:`installation <installation>` and
+:ref:`maintenance <searxng maintenance>` task.
+
+To upgrade you brand, rebase on SearXNG's master branch (on your local
+developer desktop), force push it to your remote branch.  Go to your server, do
+a force pull and run :ref:`sudo -H ./utils/searxng.sh instance update <update
+searxng>`.
+
 .. _make node.env:
 
 Node.js environment (``make node.env``)
@@ -106,36 +158,13 @@ Node.js environment (``make node.env``)
    Manager) to install latest LTS of Node.js_ locally: there is no need to
    install nvm_ or npm_ on your system.
 
-To install NVM_ and Node.js_ in once you can use :ref:`make nvm.nodejs`.
+Use ``make nvm.status`` to get the current status of you Node.js_ and nvm_ setup.
 
-.. _make nvm:
+Here is the output you will typically get on a Ubuntu 20.04 system which serves
+only a `no longer active <https://nodejs.org/en/about/releases/>`_ Release
+`Node.js v10.19.0 <https://packages.ubuntu.com/focal/nodejs>`_.
 
-NVM ``make nvm.install nvm.status``
------------------------------------
-
-Use ``make nvm.status`` to get the current status of your Node.js_ and nvm_
-setup.
-
-.. tabs::
-
-  .. group-tab:: nvm.install
-
-     .. code:: sh
-
-        $ LANG=C make nvm.install
-        INFO:  install (update) NVM at ./searxng/.nvm
-        INFO:  clone: https://github.com/nvm-sh/nvm.git
-          || Cloning into './searxng/.nvm'...
-        INFO:  checkout v0.39.4
-          || HEAD is now at 8fbf8ab v0.39.4
-
-  .. group-tab:: nvm.status (ubu2004)
-
-     Here is the output you will typically get on a Ubuntu 20.04 system which
-     serves only a `no longer active <https://nodejs.org/en/about/releases/>`_
-     Release `Node.js v10.19.0 <https://packages.ubuntu.com/focal/nodejs>`_.
-
-     .. code:: sh
+::
 
   $ make nvm.status
   INFO:  Node.js is installed at /usr/bin/node
@@ -144,11 +173,14 @@ setup.
   INFO:  npm is installed at /usr/bin/npm
   INFO:  npm is version 6.14.4
   WARN:  NVM is not installed
+  INFO:  to install NVM and Node.js (LTS) use: manage nvm install --lts
+
+To install you can also use :ref:`make nvm.nodejs`
 
 .. _make nvm.nodejs:
 
 ``make nvm.nodejs``
--------------------
+===================
 
 Install latest Node.js_ LTS locally (uses nvm_)::
 
@@ -181,29 +213,10 @@ sources of the theme need to be rebuild.  You can do that by running::
   $ make themes.all
 
 Alternatively to ``themes.all`` you can run *live builds* of the theme you are
-modify (:ref:`make themes`)::
+modify::
 
   $ LIVE_THEME=simple make run
 
-.. _make format.python:
-
-``make format.python``
-======================
-
-Format Python source code using `Black code style`_.  See ``$BLACK_OPTIONS``
-and ``$BLACK_TARGETS`` in :origin:`Makefile`.
-
-.. attention::
-
-   We stuck at Black 22.12.0, please read comment in PR `Bump black from 22.12.0
-   to 23.1.0`_
-
-.. _Bump black from 22.12.0 to 23.1.0:
-   https://github.com/searxng/searxng/pull/2159#pullrequestreview-1284094735
-
-.. _Black code style:
-   https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html
-
 .. _make clean:
 
 ``make clean``
@@ -224,36 +237,20 @@ calling ``make clean`` stop all processes using the :ref:`make install` or
 
 .. _make docs:
 
-``make docs``
-=============
-
-Target ``docs`` builds the documentation:
-
-.. code:: bash
-
-   $ make docs
-   HTML ./docs --> file://
-   DOCS      build build/docs/includes
-   ...
-   The HTML pages are in dist/docs.
-
-.. _make docs.clean:
-
-``make docs.clean docs.live``
-----------------------------------
+``make docs docs.autobuild docs.clean``
+=======================================
 
 We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute /
 Documentation <contrib docs>` section.  If you want to edit the documentation
 read our :ref:`make docs.live` section.  If you are working in your own brand,
-adjust your :ref:`settings brand`.
-
+adjust your :ref:`settings global`.
 
 .. _make docs.gh-pages:
 
 ``make docs.gh-pages``
-----------------------
+======================
 
-To deploy on github.io first adjust your :ref:`settings brand`.  For any
+To deploy on github.io first adjust your :ref:`settings global`.  For any
 further read :ref:`deploy on github.io`.
 
 .. _make test:
@@ -264,17 +261,17 @@ further read :ref:`deploy on github.io`.
 Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit``
 and ``test.robot``.  You can run tests selective, e.g.::
 
-  $ make test.pep8 test.unit test.shell
+  $ make test.pep8 test.unit test.sh
   TEST      test.pep8 OK
   ...
   TEST      test.unit OK
   ...
-  TEST      test.shell OK
+  TEST      test.sh OK
 
 .. _make test.shell:
 
 ``make test.shell``
--------------------
+===================
 
 :ref:`sh lint` / if you have changed some bash scripting run this test before
 commit.
@@ -282,7 +279,7 @@ commit.
 .. _make test.pylint:
 
 ``make test.pylint``
---------------------
+====================
 
 .. _Pylint: https://www.pylint.org/
 
@@ -292,8 +289,8 @@ found in project's root folder :origin:`.pylintrc`.
 
 .. _make search.checker:
 
-``make search.checker.{engine name}``
-=====================================
+``search.checker.{engine name}``
+================================
 
 To check all engines::
 
@@ -321,63 +318,3 @@ To filter out HTTP redirects (3xx_)::
     https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
     https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
     --
-
-.. _make themes:
-
-``make themes.*``
-=================
-
-.. sidebar:: further read
-
-   - :ref:`devquickstart`
-
-The :origin:`Makefile` targets ``make theme.*`` cover common tasks to build the
-theme(s).  The ``./manage themes.*`` command line can be used to convenient run
-common theme build tasks.
-
-.. program-output:: bash -c "cd ..; ./manage themes.help"
-
-To get live builds while modifying CSS & JS use (:ref:`make run`):
-
-.. code:: sh
-
-   $ LIVE_THEME=simple make run
-
-.. _make static.build:
-
-``make static.build.*``
-=======================
-
-.. sidebar:: further read
-
-   - :ref:`devquickstart`
-
-The :origin:`Makefile` targets ``static.build.*`` cover common tasks to build (a
-commit of) the static files.  The ``./manage static.build..*`` command line
-can be used to convenient run common build tasks of the static files.
-
-.. program-output:: bash -c "cd ..; ./manage static.help"
-
-
-.. _manage redis.help:
-
-``./manage redis.help``
-=======================
-
-The ``./manage redis.*`` command line can be used to convenient run common Redis
-tasks (:ref:`Redis developer notes`).
-
-.. program-output:: bash -c "cd ..; ./manage redis.help"
-
-
-.. _manage go.help:
-
-``./manage go.help``
-====================
-
-The ``./manage go.*`` command line can be used to convenient run common `go
-(wiki)`_ tasks.
-
-.. _go (wiki): https://en.wikipedia.org/wiki/Go_(programming_language)
-
-.. program-output:: bash -c "cd ..; ./manage go.help"

docs/dev/offline_engines.rst

@@ -1,14 +1,15 @@
+.. _offline engines:
+
 ===============
-Offline Concept
+Offline Engines
 ===============
 
 .. sidebar:: offline engines
 
    - :ref:`demo offline engine`
-   - :ref:`engine command`
    - :ref:`sql engines`
-   - :ref:`nosql engines`
-   - :py:obj:`searx.search.processors.offline`
+   - :ref:`engine command`
+   - :origin:`Redis <searx/engines/redis_server.py>`
 
 To extend the functionality of SearXNG, offline engines are going to be
 introduced.  An offline engine is an engine which does not need Internet
@@ -30,6 +31,7 @@ Programming Interface
   in advance.
 
 :py:func:`search(query, params) <searx.engines.demo_offline.searc>`
+
   Each offline engine has a function named ``search``.  This function is
   responsible to perform a search and return the results in a presentable
   format. (Where *presentable* means presentable by the selected result
@@ -67,3 +69,10 @@ administrators can set token(s) for each of the :ref:`private engines`.  If a
 query contains a valid token, then SearXNG performs the requested private
 search.  If not, requests from an offline engines return errors.
 
+
+Acknowledgement
+===============
+
+This development was sponsored by `Search and Discovery Fund
+<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ .
+

docs/dev/plugins.rst

@@ -19,6 +19,10 @@ Example plugin
    description = 'This plugin extends the suggestions with the word "example"'
    default_on = False  # disabled by default
 
+   js_dependencies = tuple()  # optional, list of static js files
+   css_dependencies = tuple()  # optional, list of static css files
+
+
    # attach callback to the post search hook
    #  request: flask request object
    #  ctx: the whole local context of the post search hook
@@ -36,7 +40,7 @@ SearXNG plugins from *The Green Web Foundation* `[ref]
 
 .. code:: bash
 
-   $ sudo utils/searxng.sh instance cmd bash -c
+   $ sudo utils/searxng.sh instance cmd bash
    (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins
 
 In the :ref:`settings.yml` activate the ``plugins:`` section and add module

docs/dev/quickstart.rst

@@ -7,16 +7,8 @@ Development Quickstart
 .. _npm: https://www.npmjs.com/
 .. _Node.js: https://nodejs.org/
 
-
-.. sidebar:: further read
-
-   - :ref:`makefile`
-   - :ref:`buildhosts`
-
-SearXNG loves developers; Developers do not need to worry about tool chains, the
-usual developer tasks can be comfortably executed via :ref:`make <makefile>`.
-
-Don't hesitate, just clone SearXNG's sources and start hacking right now ..
+SearXNG loves developers, just clone and start hacking.  All the rest is done for
+you simply by using :ref:`make <makefile>`.
 
 .. code:: bash
 
@@ -26,23 +18,25 @@ Here is how a minimal workflow looks like:
 
 1. *start* hacking
 2. *run* your code: :ref:`make run`
-3. *format & test* your code: :ref:`make format.python` and :ref:`make test`
+3. *test* your code: :ref:`make test`
 
 If you think at some point something fails, go back to *start*.  Otherwise,
 choose a meaningful commit message and we are happy to receive your pull
 request. To not end in *wild west* we have some directives, please pay attention
 to our ":ref:`how to contribute`" guideline.
 
-.. sidebar:: further read
+If you implement themes, you will need to setup a :ref:`make node.env` once:
 
-   - :ref:`make nvm`
-   - :ref:`make themes`
+.. code:: bash
 
-If you implement themes, you will need to setup a :ref:`Node.js environment
-<make node.env>`: ``make node.env``
+   make node.env
 
 Before you call *make run* (2.), you need to compile the modified styles and
-JavaScript: ``make themes.all``
+JavaScript:
+
+.. code:: bash
+
+   make themes.all
 
 Alternatively you can also compile selective the theme you have modified,
 e.g. the *simple* theme.
@@ -55,10 +49,6 @@ e.g. the *simple* theme.
 
    To get live builds while modifying CSS & JS use: ``LIVE_THEME=simple make run``
 
-.. sidebar:: further read
-
-   - :ref:`make static.build`
-
 If you finished your *tests* you can start to commit your changes.  To separate
 the modified source code from the build products first run:
 
@@ -67,14 +57,14 @@ the modified source code from the build products first run:
    make static.build.restore
 
 This will restore the old build products and only your changes of the code
-remain in the working tree which can now be added & committed.  When all sources
-are committed, you can commit the build products simply by:
+remain in the working tree which can now be added & commited.  When all sources
+are commited, you can commit the build products simply by:
 
 .. code:: bash
 
    make static.build.commit
 
-Committing the build products should be the last step, just before you send us
+Commiting the build products should be the last step, just before you send us
 your PR.  There is also a make target to rewind this last build commit:
 
 .. code:: bash

docs/dev/reST.rst

@@ -37,7 +37,7 @@ docs.live <make docs.live>` to build HTML while editing.
    - DOT_, `Graphviz's dot`_, Graphviz_
 
 
-.. contents::
+.. contents:: Contents
    :depth: 3
    :local:
    :backlinks: entry
@@ -235,7 +235,7 @@ To refer anchors use the `ref role`_ markup:
 .. admonition:: ``:ref:`` role
    :class: rst-example
 
-   Visit chapter :ref:`reST anchor`.  Or set hyperlink text manually :ref:`foo
+   Visist chapter :ref:`reST anchor`.  Or set hyperlink text manually :ref:`foo
    bar <reST anchor>`.
 
 .. _reST ordinary ref:
@@ -292,7 +292,7 @@ content becomes smart.
    files & folders origin     :origin:`docs/dev/reST.rst`        ``:origin:`docs/dev/reST.rst```
    pull request               :pull:`4`                          ``:pull:`4```
    patch                      :patch:`af2cae6`                   ``:patch:`af2cae6```
-   PyPi package               :pypi:`httpx`                      ``:pypi:`httpx```
+   PyPi package               :pypi:`searx`                      ``:pypi:`searx```
    manual page man            :man:`bash`                        ``:man:`bash```
    intersphinx_
    --------------------------------------------------------------------------------------------------