Compare commits

..

1 Commits

Author SHA1 Message Date
Markus Heiser e0a3dee3bf [POC] limiter: change PING of link_token mehtod from CSS to <img>
while PR #2357 [1] was being implemented the question came up:

    would be better to change the PING resource from CSS to an image so that
    some terminal based browser may still able to pass the test [1]

This patch implements a POC in where a <img src=token> tag is loaded instaed a
CSS.

To test this patch activate limiter and link_token method [3] and start a
developer instance::

    make run

In your terminal browser open http://127.0.0.1:8888/search?q=foo

If the browser is suitable for the link_token method, it loads the image and the
following messages appear::

    DEBUG   searx.botdetection.limiter    : OK 127.0.0.1/32: /clientft61aak7fzyu6o6v.svg ...
    DEBUG   searx.botdetection.link_token : token is valid --> True
    DEBUG   searx.botdetection.link_token : store ping_key for (client) network 127.0.0.1/32 (IP 127.0.0.1) -> SearXNG_limiter.ping[...]

Browsers that do not load images will be blocked: If you try by example::

    lynx http://127.0.0.1:8888/search?q=foo

you will see a WARNING message like::

    WARNING searx.botdetection.link_token : missing ping (IP: 127.0.0.1/32) / request: SearXNG_limiter.ping[...]

----

[1] 80aaef6c95
[2] https://github.com/searxng/searxng/pull/2357#issuecomment-1574898834
[3] activate limiter and link_token method

```diff
diff --git a/searx/botdetection/limiter.toml b/searx/botdetection/limiter.toml
index 71a231e8f..7e1dba755 100644
--- a/searx/botdetection/limiter.toml
+++ b/searx/botdetection/limiter.toml
@@ -17,6 +17,6 @@ ipv6_prefix = 48
 filter_link_local = false

 # acrivate link_token method in the ip_limit method
-link_token = false
+link_token = true

diff --git a/searx/settings.yml b/searx/settings.yml
index a82a3432d..e7b983afc 100644
--- a/searx/settings.yml
+++ b/searx/settings.yml
@@ -73,7 +73,7 @@ server:
   # public URL of the instance, to ensure correct inbound links. Is overwritten
   # by ${SEARXNG_URL}.
   base_url: false  # "http://example.com/location"
-  limiter: false  # rate limit the number of request on the instance, block some bots
+  limiter: true  # rate limit the number of request on the instance, block some bots

   # If your instance owns a /etc/searxng/settings.yml file, then set the following
   # values there.
```

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-03 18:49:21 +02:00
688 changed files with 86800 additions and 182243 deletions

View File

@ -16,9 +16,6 @@ max_line_length = 119
[*.html] [*.html]
indent_size = 4 indent_size = 4
[*.js]
indent_size = 2
[*.json] [*.json]
indent_size = 4 indent_size = 4
insert_final_newline = ignore insert_final_newline = ignore

View File

@ -20,7 +20,7 @@ assignees: ''
**How can SearXNG fetch the information from this engine?** **How can SearXNG fetch the information from this engine?**
<!-- List API URL, example code (using the correct markdown) and more <!-- List API URL, example code (using the correct markdown) and more
that could be useful for the developers in order to implement this engine. that could be useful for the developers in order to implement this engine.
If you don't know what to write, let this part blank. --> If you don't know what to write, let this part blank.>
**Applicable category of this engine** **Applicable category of this engine**
<!-- Where should this new engine fit in SearXNG? Current categories in SearXNG: <!-- Where should this new engine fit in SearXNG? Current categories in SearXNG:

View File

@ -1,4 +1,4 @@
# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file # https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
version: 2 version: 2
updates: updates:
- package-ecosystem: "pip" - package-ecosystem: "pip"
@ -8,9 +8,6 @@ updates:
day: "friday" day: "friday"
open-pull-requests-limit: 5 open-pull-requests-limit: 5
target-branch: "master" target-branch: "master"
commit-message:
prefix: "[upd] pypi:"
- package-ecosystem: "npm" - package-ecosystem: "npm"
directory: "/searx/static/themes/simple" directory: "/searx/static/themes/simple"
schedule: schedule:
@ -18,5 +15,3 @@ updates:
day: "friday" day: "friday"
open-pull-requests-limit: 5 open-pull-requests-limit: 5
target-branch: "master" target-branch: "master"
commit-message:
prefix: "[upd] npm:"

View File

@ -1,31 +0,0 @@
name: "Checker"
on:
schedule:
- cron: "0 4 * * 5"
workflow_dispatch:
jobs:
checker:
name: Checker
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Ubuntu packages
run: |
sudo ./utils/searxng.sh install packages
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
architecture: 'x64'
- name: Install Python dependencies
run: |
make V=1 install
- name: Checker
run: |
make search.checker

View File

@ -7,7 +7,7 @@ on:
jobs: jobs:
updateData: updateData:
name: Update data - ${{ matrix.fetch }} name: Update data - ${{ matrix.fetch }}
runs-on: ubuntu-24.04 runs-on: ubuntu-20.04
if: ${{ github.repository_owner == 'searxng'}} if: ${{ github.repository_owner == 'searxng'}}
strategy: strategy:
fail-fast: false fail-fast: false
@ -22,16 +22,16 @@ jobs:
- update_engine_descriptions.py - update_engine_descriptions.py
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v2
- name: Install Ubuntu packages - name: Install Ubuntu packages
run: | run: |
sudo ./utils/searxng.sh install packages sudo ./utils/searxng.sh install packages
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v5 uses: actions/setup-python@v2
with: with:
python-version: '3.12' python-version: '3.9'
architecture: 'x64' architecture: 'x64'
- name: Install Python dependencies - name: Install Python dependencies
@ -46,18 +46,18 @@ jobs:
- name: Create Pull Request - name: Create Pull Request
id: cpr id: cpr
uses: peter-evans/create-pull-request@v6 uses: peter-evans/create-pull-request@v3
with: with:
commit-message: '[data] update searx.data - ${{ matrix.fetch }}' commit-message: Update searx.data - ${{ matrix.fetch }}
committer: searxng-bot <noreply@github.com> committer: searxng-bot <noreply@github.com>
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com> author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
signoff: false signoff: false
branch: update_data_${{ matrix.fetch }} branch: update_data_${{ matrix.fetch }}
delete-branch: true delete-branch: true
draft: false draft: false
title: '[data] update searx.data - ${{ matrix.fetch }}' title: 'Update searx.data - ${{ matrix.fetch }}'
body: | body: |
update searx.data - ${{ matrix.fetch }} Update searx.data - ${{ matrix.fetch }}
labels: | labels: |
data data

View File

@ -16,16 +16,16 @@ jobs:
strategy: strategy:
matrix: matrix:
os: [ubuntu-20.04] os: [ubuntu-20.04]
python-version: ["3.9", "3.10", "3.11", "3.12",] python-version: ["3.8", "3.9", "3.10", "3.11"]
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v2
- name: Install Ubuntu packages - name: Install Ubuntu packages
run: | run: |
sudo ./utils/searxng.sh install packages sudo ./utils/searxng.sh install packages
sudo apt install firefox sudo apt install firefox
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v5 uses: actions/setup-python@v4
with: with:
python-version: ${{ matrix.python-version }} python-version: ${{ matrix.python-version }}
architecture: 'x64' architecture: 'x64'
@ -45,19 +45,27 @@ jobs:
make V=1 gecko.driver make V=1 gecko.driver
- name: Run tests - name: Run tests
run: make V=1 ci.test run: make V=1 ci.test
- name: Test coverage
run: make V=1 test.coverage
- name: Store coverage result
uses: actions/upload-artifact@v2
with:
name: coverage-${{ matrix.python-version }}
path: coverage/
retention-days: 60
themes: themes:
name: Themes name: Themes
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v2
- name: Install Ubuntu packages - name: Install Ubuntu packages
run: sudo ./utils/searxng.sh install buildhost run: sudo ./utils/searxng.sh install buildhost
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v5 uses: actions/setup-python@v4
with: with:
python-version: '3.12' python-version: '3.9'
architecture: 'x64' architecture: 'x64'
- name: Cache Python dependencies - name: Cache Python dependencies
id: cache-python id: cache-python
@ -67,7 +75,7 @@ jobs:
./local ./local
./.nvm ./.nvm
./node_modules ./node_modules
key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }} key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
- name: Install node dependencies - name: Install node dependencies
run: make V=1 node.env run: make V=1 node.env
- name: Build themes - name: Build themes
@ -80,16 +88,16 @@ jobs:
contents: write # for JamesIves/github-pages-deploy-action to push changes in repo contents: write # for JamesIves/github-pages-deploy-action to push changes in repo
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v2
with: with:
fetch-depth: '0' fetch-depth: '0'
persist-credentials: false persist-credentials: false
- name: Install Ubuntu packages - name: Install Ubuntu packages
run: sudo ./utils/searxng.sh install buildhost run: sudo ./utils/searxng.sh install buildhost
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v5 uses: actions/setup-python@v4
with: with:
python-version: '3.12' python-version: '3.9'
architecture: 'x64' architecture: 'x64'
- name: Cache Python dependencies - name: Cache Python dependencies
id: cache-python id: cache-python
@ -99,7 +107,7 @@ jobs:
./local ./local
./.nvm ./.nvm
./node_modules ./node_modules
key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }} key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
- name: Build documentation - name: Build documentation
run: | run: |
make V=1 docs.clean docs.html make V=1 docs.clean docs.html
@ -112,7 +120,7 @@ jobs:
FOLDER: dist/docs FOLDER: dist/docs
CLEAN: true # Automatically remove deleted files from the deploy branch CLEAN: true # Automatically remove deleted files from the deploy branch
SINGLE_COMMIT: True SINGLE_COMMIT: True
COMMIT_MESSAGE: '[doc] build from commit ${{ github.sha }}' COMMIT_MESSAGE: build from commit ${{ github.sha }}
babel: babel:
name: Update translations branch name: Update translations branch
@ -126,14 +134,14 @@ jobs:
contents: write # for make V=1 weblate.push.translations contents: write # for make V=1 weblate.push.translations
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v2
with: with:
fetch-depth: '0' fetch-depth: '0'
token: ${{ secrets.WEBLATE_GITHUB_TOKEN }} token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v5 uses: actions/setup-python@v4
with: with:
python-version: '3.12' python-version: '3.9'
architecture: 'x64' architecture: 'x64'
- name: Cache Python dependencies - name: Cache Python dependencies
id: cache-python id: cache-python
@ -143,7 +151,7 @@ jobs:
./local ./local
./.nvm ./.nvm
./node_modules ./node_modules
key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }} key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
- name: weblate & git setup - name: weblate & git setup
env: env:
WEBLATE_CONFIG: ${{ secrets.WEBLATE_CONFIG }} WEBLATE_CONFIG: ${{ secrets.WEBLATE_CONFIG }}
@ -155,6 +163,7 @@ jobs:
- name: Update transations - name: Update transations
id: update id: update
run: | run: |
git restore utils/brand.env
make V=1 weblate.push.translations make V=1 weblate.push.translations
dockers: dockers:
@ -170,14 +179,14 @@ jobs:
steps: steps:
- name: Checkout - name: Checkout
if: env.DOCKERHUB_USERNAME != null if: env.DOCKERHUB_USERNAME != null
uses: actions/checkout@v4 uses: actions/checkout@v2
with: with:
# make sure "make docker.push" can get the git history # make sure "make docker.push" can get the git history
fetch-depth: '0' fetch-depth: '0'
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v5 uses: actions/setup-python@v4
with: with:
python-version: '3.12' python-version: '3.9'
architecture: 'x64' architecture: 'x64'
- name: Cache Python dependencies - name: Cache Python dependencies
id: cache-python id: cache-python
@ -187,7 +196,7 @@ jobs:
./local ./local
./.nvm ./.nvm
./node_modules ./node_modules
key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }} key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
- name: Set up QEMU - name: Set up QEMU
if: env.DOCKERHUB_USERNAME != null if: env.DOCKERHUB_USERNAME != null
uses: docker/setup-qemu-action@v1 uses: docker/setup-qemu-action@v1

View File

@ -10,7 +10,7 @@ jobs:
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v2
- name: Run Trivy vulnerability scanner - name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master uses: aquasecurity/trivy-action@master

View File

@ -6,19 +6,19 @@ on:
jobs: jobs:
babel: babel:
name: "create PR for additions from weblate" name: "create PR for additons from weblate"
runs-on: ubuntu-20.04 runs-on: ubuntu-20.04
if: ${{ github.repository_owner == 'searxng' && github.ref == 'refs/heads/master' }} if: ${{ github.repository_owner == 'searxng' && github.ref == 'refs/heads/master' }}
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v2
with: with:
fetch-depth: '0' fetch-depth: '0'
token: ${{ secrets.WEBLATE_GITHUB_TOKEN }} token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
- name: Set up Python - name: Set up Python
uses: actions/setup-python@v5 uses: actions/setup-python@v4
with: with:
python-version: '3.12' python-version: '3.9'
architecture: 'x64' architecture: 'x64'
- name: Cache Python dependencies - name: Cache Python dependencies
id: cache-python id: cache-python
@ -28,7 +28,7 @@ jobs:
./local ./local
./.nvm ./.nvm
./node_modules ./node_modules
key: python-ubuntu-20.04-3.12-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }} key: python-ubuntu-20.04-3.9-${{ hashFiles('requirements*.txt', 'setup.py','.nvmrc', 'package.json') }}
- name: weblate & git setup - name: weblate & git setup
env: env:
WEBLATE_CONFIG: ${{ secrets.WEBLATE_CONFIG }} WEBLATE_CONFIG: ${{ secrets.WEBLATE_CONFIG }}
@ -45,15 +45,15 @@ jobs:
uses: peter-evans/create-pull-request@v3 uses: peter-evans/create-pull-request@v3
with: with:
token: ${{ secrets.WEBLATE_GITHUB_TOKEN }} token: ${{ secrets.WEBLATE_GITHUB_TOKEN }}
commit-message: '[l10n] update translations from Weblate' commit-message: Update translations
committer: searxng-bot <searxng-bot@users.noreply.github.com> committer: searxng-bot <searxng-bot@users.noreply.github.com>
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com> author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
signoff: false signoff: false
branch: translations_update branch: translations_update
delete-branch: true delete-branch: true
draft: false draft: false
title: '[l10n] update translations from Weblate' title: 'Update translations'
body: | body: |
update translations from Weblate Update translations
labels: | labels: |
translation translation

3
.gitignore vendored
View File

@ -23,6 +23,3 @@ gh-pages/
.idea/ .idea/
searx/version_frozen.py searx/version_frozen.py
.dir-locals.el
.python-version

2
.nvmrc
View File

@ -1 +1 @@
v20.10 v16.15.1

View File

@ -1,4 +1,4 @@
# -*- coding: utf-8; mode: conf-unix -*- # -*- coding: utf-8; mode: conf -*-
# lint Python modules using external checkers. # lint Python modules using external checkers.
# #
# This is the main checker controlling the other ones and the reports # This is the main checker controlling the other ones and the reports
@ -27,7 +27,7 @@ ignore-patterns=
#init-hook= #init-hook=
# Use multiple processes to speed up Pylint. # Use multiple processes to speed up Pylint.
jobs=0 jobs=1
# List of plugins (as comma separated values of python modules names) to load, # List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers. # usually to register additional checkers.
@ -338,7 +338,6 @@ valid-metaclass-classmethod-first-arg=mcs
# Maximum number of arguments for function / method # Maximum number of arguments for function / method
max-args=8 max-args=8
max-positional-arguments=14
# Maximum number of attributes for a class (see R0902). # Maximum number of attributes for a class (see R0902).
max-attributes=20 max-attributes=20

View File

@ -1,2 +0,0 @@
python 3.12.0
shellcheck 0.9.0

View File

@ -168,9 +168,3 @@ features or generally made searx better:
- Milad Laly @Milad-Laly - Milad Laly @Milad-Laly
- @llmII - @llmII
- @blob42 `<https://blob42.xyz>`_ - @blob42 `<https://blob42.xyz>`_
- Paolo Basso `<https://github.com/paolobasso99>`
- Bernie Huang `<https://github.com/BernieHuang2008>`
- Austin Olacsi `<https://github.com/Austin-Olacsi>`
- @micsthepick
- Daniel Kukula `<https://github.com/dkuku>`
- Patrick Evans `https://github.com/holysoles`

View File

@ -2,7 +2,7 @@
## Resources in the documentation ## Resources in the documentation
* [Development quickstart](https://docs.searxng.org/dev/quickstart.html) * [Development quickstart](https://docs.searxng.org/dev/contribution_guide.html)
* [Contribution guide](https://docs.searxng.org/dev/contribution_guide.html) * [Contribution guide](https://docs.searxng.org/dev/contribution_guide.html)
## Submitting PRs ## Submitting PRs

View File

@ -1,4 +1,4 @@
FROM alpine:3.20 FROM alpine:3.18
ENTRYPOINT ["/sbin/tini","--","/usr/local/searxng/dockerfiles/docker-entrypoint.sh"] ENTRYPOINT ["/sbin/tini","--","/usr/local/searxng/dockerfiles/docker-entrypoint.sh"]
EXPOSE 8080 EXPOSE 8080
VOLUME /etc/searxng VOLUME /etc/searxng
@ -15,9 +15,7 @@ ENV INSTANCE_NAME=searxng \
MORTY_KEY= \ MORTY_KEY= \
MORTY_URL= \ MORTY_URL= \
SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml \ SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml \
UWSGI_SETTINGS_PATH=/etc/searxng/uwsgi.ini \ UWSGI_SETTINGS_PATH=/etc/searxng/uwsgi.ini
UWSGI_WORKERS=%k \
UWSGI_THREADS=4
WORKDIR /usr/local/searxng WORKDIR /usr/local/searxng
@ -35,6 +33,7 @@ RUN apk add --no-cache -t build-dependencies \
git \ git \
&& apk add --no-cache \ && apk add --no-cache \
ca-certificates \ ca-certificates \
su-exec \
python3 \ python3 \
py3-pip \ py3-pip \
libxml2 \ libxml2 \
@ -44,7 +43,7 @@ RUN apk add --no-cache -t build-dependencies \
uwsgi \ uwsgi \
uwsgi-python3 \ uwsgi-python3 \
brotli \ brotli \
&& pip3 install --break-system-packages --no-cache -r requirements.txt \ && pip3 install --no-cache -r requirements.txt \
&& apk del build-dependencies \ && apk del build-dependencies \
&& rm -rf /root/.cache && rm -rf /root/.cache

View File

@ -44,10 +44,10 @@ lxc.clean:
PHONY += search.checker search.checker.% PHONY += search.checker search.checker.%
search.checker: install search.checker: install
$(Q)./manage pyenv.cmd searxng-checker -v $(Q)./manage pyenv.cmd searx-checker -v
search.checker.%: install search.checker.%: install
$(Q)./manage pyenv.cmd searxng-checker -v "$(subst _, ,$(patsubst search.checker.%,%,$@))" $(Q)./manage pyenv.cmd searx-checker -v "$(subst _, ,$(patsubst search.checker.%,%,$@))"
PHONY += test ci.test test.shell PHONY += test ci.test test.shell
ci.test: test.yamllint test.black test.pyright test.pylint test.unit test.robot test.rst test.pybabel ci.test: test.yamllint test.black test.pyright test.pylint test.unit test.robot test.rst test.pybabel
@ -56,32 +56,34 @@ test.shell:
$(Q)shellcheck -x -s dash \ $(Q)shellcheck -x -s dash \
dockerfiles/docker-entrypoint.sh dockerfiles/docker-entrypoint.sh
$(Q)shellcheck -x -s bash \ $(Q)shellcheck -x -s bash \
utils/brand.sh \ utils/brand.env \
$(MTOOLS) \ $(MTOOLS) \
utils/lib.sh \ utils/lib.sh \
utils/lib_sxng*.sh \
utils/lib_go.sh \
utils/lib_nvm.sh \ utils/lib_nvm.sh \
utils/lib_static.sh \
utils/lib_go.sh \
utils/lib_redis.sh \ utils/lib_redis.sh \
utils/searxng.sh \
utils/lxc.sh \
utils/lxc-searxng.env \
utils/searx.sh \
utils/filtron.sh \ utils/filtron.sh \
utils/morty.sh utils/searx.sh \
utils/searxng.sh \
utils/morty.sh \
utils/lxc.sh \
utils/lxc-searxng.env
$(Q)$(MTOOLS) build_msg TEST "$@ OK" $(Q)$(MTOOLS) build_msg TEST "$@ OK"
# wrap ./manage script # wrap ./manage script
MANAGE += buildenv
MANAGE += weblate.translations.commit weblate.push.translations MANAGE += weblate.translations.commit weblate.push.translations
MANAGE += data.all data.traits data.useragents data.locales MANAGE += data.all data.traits data.useragents
MANAGE += docs.html docs.live docs.gh-pages docs.prebuild docs.clean MANAGE += docs.html docs.live docs.gh-pages docs.prebuild docs.clean
MANAGE += docker.build docker.push docker.buildx MANAGE += docker.build docker.push docker.buildx
MANAGE += gecko.driver MANAGE += gecko.driver
MANAGE += node.env node.env.dev node.clean MANAGE += node.env node.env.dev node.clean
MANAGE += py.build py.clean MANAGE += py.build py.clean
MANAGE += pyenv pyenv.install pyenv.uninstall MANAGE += pyenv pyenv.install pyenv.uninstall
MANAGE += pypi.upload pypi.upload.test
MANAGE += format.python MANAGE += format.python
MANAGE += test.yamllint test.pylint test.pyright test.black test.pybabel test.unit test.coverage test.robot test.rst test.clean MANAGE += test.yamllint test.pylint test.pyright test.black test.pybabel test.unit test.coverage test.robot test.rst test.clean
MANAGE += themes.all themes.simple themes.simple.test pygments.less MANAGE += themes.all themes.simple themes.simple.test pygments.less

View File

@ -12,11 +12,11 @@
## How to test this PR locally? ## How to test this PR locally?
<!-- commands to run the tests or instructions to test the changes --> <!-- commands to run the tests or instructions to test the changes-->
## Author's checklist ## Author's checklist
<!-- additional notes for reviewers --> <!-- additional notes for reviewiers -->
## Related issues ## Related issues

View File

@ -66,7 +66,7 @@ A user_, admin_ and developer_ handbook is available on the homepage_.
Contact Contact
======= =======
Ask questions or chat with the SearXNG community (this not a chatbot) on Ask questions or just chat about SearXNG on
IRC IRC
`#searxng on libera.chat <https://web.libera.chat/?channel=#searxng>`_ `#searxng on libera.chat <https://web.libera.chat/?channel=#searxng>`_
@ -75,42 +75,76 @@ IRC
Matrix Matrix
`#searxng:matrix.org <https://matrix.to/#/#searxng:matrix.org>`_ `#searxng:matrix.org <https://matrix.to/#/#searxng:matrix.org>`_
Differences to searx
====================
SearXNG is a fork of `searx`_, with notable changes:
.. _searx: https://github.com/searx/searx
User experience
---------------
- Reworked (and still simple) theme:
* Usable on desktop, tablet and mobile.
* Light and dark versions (available in the preferences).
* Right-to-left language support.
* `Screenshots <https://dev.searxng.org/screenshots.html>`_
- The translations are up to date, you can contribute on `Weblate`_
- The preferences page has been updated:
* Browse which engines are reliable or not.
* Engines are grouped inside each tab.
* Each engine has a description.
- Thanks to the anonymous metrics, it is easier to report malfunctioning engines,
so they get fixed quicker
- `Turn off metrics on the server
<https://docs.searxng.org/admin/engines/settings.html#general>`_ if you don't want them recorded.
- Administrators can `block and/or replace the URLs in the search results
<https://github.com/searxng/searxng/blob/5c1c0817c3996c5670a545d05831d234d21e6217/searx/settings.yml#L191-L199>`_
Setup Setup
===== -----
- A well maintained `Docker image`_, also built for ARM64 and ARM/v7 - No need for `Morty`_ to proxy images, even on a public instance.
architectures. - No need for `Filtron`_ to block bots, as there is now a built-in `limiter`_.
- Alternatively there are *up to date* `installation scripts`_. - A well maintained `Docker image`_, now also built for ARM64 and ARM/v7 architectures.
- For individual setup consult our detailed `Step by step`_ instructions. (Alternatively there are up to date installation scripts.)
- To fine-tune your instance, take a look at the `Administrator documentation`_.
.. _Administrator documentation: https://docs.searxng.org/admin/index.html
.. _Step by step: https://docs.searxng.org/admin/installation-searxng.html
.. _installation scripts: https://docs.searxng.org/admin/installation-scripts.html
.. _Docker image: https://github.com/searxng/searxng-docker .. _Docker image: https://github.com/searxng/searxng-docker
Contributing
------------
- Readable debug log.
- Contributing is easier, thanks to the `Development Quickstart`_ guide.
- A lot of code cleanup and bugfixes.
- Up to date list dependencies.
.. _Morty: https://github.com/asciimoo/morty
.. _Filtron: https://github.com/searxng/filtron
.. _limiter: https://docs.searxng.org/src/searx.plugins.limiter.html
.. _Weblate: https://translate.codeberg.org/projects/searxng/searxng/
.. _Development Quickstart: https://docs.searxng.org/dev/quickstart.html
Translations Translations
============ ============
.. _Weblate: https://translate.codeberg.org/projects/searxng/searxng/
Help translate SearXNG at `Weblate`_ Help translate SearXNG at `Weblate`_
.. figure:: https://translate.codeberg.org/widgets/searxng/-/multi-auto.svg .. figure:: https://translate.codeberg.org/widgets/searxng/-/multi-auto.svg
:target: https://translate.codeberg.org/projects/searxng/ :target: https://translate.codeberg.org/projects/searxng/
Contributing
============
.. _development quickstart: https://docs.searxng.org/dev/quickstart.html
.. _developer documentation: https://docs.searxng.org/dev/index.html
Are you a developer? Have a look at our `development quickstart`_ guide, it's
very easy to contribute. Additionally we have a `developer documentation`_.
Codespaces Codespaces
========== ==========
@ -121,9 +155,9 @@ You can contribute from your browser using `GitHub Codespaces`_:
- Click on the ``Codespaces`` tab instead of ``Local`` - Click on the ``Codespaces`` tab instead of ``Local``
- Click on ``Create codespace on master`` - Click on ``Create codespace on master``
- VSCode is going to start in the browser - VSCode is going to start in the browser
- Wait for ``git pull && make install`` to appear and then disappear - Wait for ``git pull && make install`` to appears and then to disapear
- You have `120 hours per month`_ (see also your `list of existing Codespaces`_) - You have `120 hours per month`_ (see also your `list of existing Codespaces`_)
- You can start SearXNG using ``make run`` in the terminal or by pressing ``Ctrl+Shift+B`` - You can start SearXNG using ``make run`` in the terminal or by pressing ``Ctrl+Shift+B``.
.. _GitHub Codespaces: https://docs.github.com/en/codespaces/overview .. _GitHub Codespaces: https://docs.github.com/en/codespaces/overview
.. _120 hours per month: https://github.com/settings/billing .. _120 hours per month: https://github.com/settings/billing

View File

@ -175,4 +175,4 @@ unset MORTY_KEY
# Start uwsgi # Start uwsgi
printf 'Listen on %s\n' "${BIND_ADDRESS}" printf 'Listen on %s\n' "${BIND_ADDRESS}"
exec uwsgi --master --uid searxng --gid searxng --http-socket "${BIND_ADDRESS}" "${UWSGI_SETTINGS_PATH}" exec su-exec searxng:searxng uwsgi --master --http-socket "${BIND_ADDRESS}" "${UWSGI_SETTINGS_PATH}"

View File

@ -4,12 +4,8 @@ uid = searxng
gid = searxng gid = searxng
# Number of workers (usually CPU count) # Number of workers (usually CPU count)
# default value: %k (= number of CPU core, see Dockerfile) workers = %k
workers = $(UWSGI_WORKERS) threads = 4
# Number of threads per worker
# default value: 4 (see Dockerfile)
threads = $(UWSGI_THREADS)
# The right granted on the created socket # The right granted on the created socket
chmod-socket = 666 chmod-socket = 666
@ -42,13 +38,12 @@ buffer-size = 8192
# See https://github.com/searx/searx-docker/issues/24 # See https://github.com/searx/searx-docker/issues/24
add-header = Connection: close add-header = Connection: close
# Follow SIGTERM convention
# See https://github.com/searxng/searxng/issues/3427
die-on-term
# uwsgi serves the static files # uwsgi serves the static files
# expires set to one year since there are hashes
static-map = /static=/usr/local/searxng/searx/static static-map = /static=/usr/local/searxng/searx/static
# expires set to one day static-expires = /* 31557600
static-expires = /* 86400
static-gzip-all = True static-gzip-all = True
offload-threads = %k offload-threads = %k
# Cache
cache2 = name=searxngcache,items=2000,blocks=2000,blocksize=4096,bitmap=1

View File

@ -4,4 +4,3 @@ stylesheet = searxng.css
[options] [options]
touch_icon = touch_icon =
globaltoc_maxdepth = 5

View File

@ -69,6 +69,10 @@ Sample response
{ {
"enabled": true, "enabled": true,
"name": "HTTPS rewrite" "name": "HTTPS rewrite"
},
{
"enabled": false,
"name": "Vim-like hotkeys"
} }
], ],
"safe_search": 0 "safe_search": 0
@ -84,9 +88,9 @@ HTML of the site. URL of the SearXNG instance and values are customizable.
.. code:: html .. code:: html
<form method="post" action="https://example.org/"> <form method="post" action="https://example.org/">
<!-- search --> <input type="text" name="q"> <!-- search --> <input type="text" name="q" />
<!-- categories --> <input type="hidden" name="categories" value="general,social media"> <!-- categories --> <input type="hidden" name="categories" value="general,social media" />
<!-- language --> <input type="hidden" name="lang" value="all"> <!-- language --> <input type="hidden" name="lang" value="all" />
<!-- locale --> <input type="hidden" name="locale" value="en"> <!-- locale --> <input type="hidden" name="locale" value="en" />
<!-- date filter --> <input type="hidden" name="time_range" value="month"> <!-- date filter --> <input type="hidden" name="time_range" value="month" />
</form> </form>

View File

@ -4,36 +4,28 @@
Buildhosts Buildhosts
========== ==========
.. contents:: .. sidebar:: This article needs some work
If you have any contribution send us your :pull:`PR <../pulls>`, see
:ref:`how to contribute`.
.. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
To get best results from build, it's recommend to install additional packages on To get best results from build, it's recommend to install additional packages
build hosts (see :ref:`searxng.sh`). on build hosts (see :ref:`searxng.sh`).::
.. _searxng.sh install buildhost: sudo -H ./utils/searxng.sh install buildhost
Build and Development tools This will install packages needed by searx:
===========================
To Install tools used by build and development tasks in once:
.. tabs::
.. group-tab:: SearXNG's development tools
.. code:: sh
$ sudo -H ./utils/searxng.sh install buildhost
This will install packages needed by SearXNG:
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst .. kernel-include:: $DOCS_BUILD/includes/searxng.rst
:start-after: START distro-packages :start-after: START distro-packages
:end-before: END distro-packages :end-before: END distro-packages
and packages needed to build documentation and run tests: and packages needed to build docuemtation and run tests:
.. kernel-include:: $DOCS_BUILD/includes/searxng.rst .. kernel-include:: $DOCS_BUILD/includes/searxng.rst
:start-after: START build-packages :start-after: START build-packages
@ -81,7 +73,7 @@ If your docs build (``make docs.html``) shows warnings like this::
display), check the imgmath_latex setting display), check the imgmath_latex setting
you need to install additional packages on your build host, to get better HTML you need to install additional packages on your build host, to get better HTML
output (:ref:`install buildhost <searxng.sh install buildhost>`). output.
.. tabs:: .. tabs::
@ -101,7 +93,7 @@ output (:ref:`install buildhost <searxng.sh install buildhost>`).
.. code-block:: sh .. code-block:: sh
$ sudo dnf install graphviz graphviz-gd ImageMagick texlive-xetex-bin librsvg2-tools $ sudo dnf install graphviz graphviz-gd texlive-xetex-bin librsvg2-tools
For PDF output you also need: For PDF output you also need:
@ -126,7 +118,8 @@ For PDF output you also need:
$ sudo dnf install \ $ sudo dnf install \
texlive-collection-fontsrecommended texlive-collection-latex \ texlive-collection-fontsrecommended texlive-collection-latex \
dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts \
ImageMagick
.. _sh lint: .. _sh lint:
@ -135,8 +128,7 @@ Lint shell scripts
.. _ShellCheck: https://github.com/koalaman/shellcheck .. _ShellCheck: https://github.com/koalaman/shellcheck
To lint shell scripts we use ShellCheck_ - a shell script static analysis tool To lint shell scripts, we use ShellCheck_ - a shell script static analysis tool.
(:ref:`install buildhost <searxng.sh install buildhost>`).
.. SNIP sh lint requirements .. SNIP sh lint requirements

View File

@ -0,0 +1,79 @@
.. _engine command:
====================
Command Line Engines
====================
.. sidebar:: info
- :origin:`command.py <searx/engines/command.py>`
- :ref:`offline engines`
With *command engines* administrators can run engines to integrate arbitrary
shell commands.
When creating and enabling a ``command`` engine on a public instance, you must
be careful to avoid leaking private data. The easiest solution is to limit the
access by setting ``tokens`` as described in section :ref:`private engines`.
The engine base is flexible. Only your imagination can limit the power of this
engine (and maybe security concerns). The following options are available:
``command``:
A comma separated list of the elements of the command. A special token
``{{QUERY}}`` tells where to put the search terms of the user. Example:
.. code:: yaml
['ls', '-l', '-h', '{{QUERY}}']
``delimiter``:
A mapping containing a delimiter ``char`` and the *titles* of each element in
``keys``.
``parse_regex``:
A dict containing the regular expressions for each result key.
``query_type``:
The expected type of user search terms. Possible values: ``path`` and
``enum``.
``path``:
Checks if the user provided path is inside the working directory. If not,
the query is not executed.
``enum``:
Is a list of allowed search terms. If the user submits something which is
not included in the list, the query returns an error.
``query_enum``:
A list containing allowed search terms if ``query_type`` is set to ``enum``.
``working_dir``:
The directory where the command has to be executed. Default: ``./``
``result_separator``:
The character that separates results. Default: ``\n``
The example engine below can be used to find files with a specific name in the
configured working directory:
.. code:: yaml
- name: find
engine: command
command: ['find', '.', '-name', '{{QUERY}}']
query_type: path
shortcut: fnd
delimiter:
chars: ' '
keys: ['line']
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View File

@ -0,0 +1,26 @@
.. _engines and settings:
==================
Engines & Settings
==================
.. sidebar:: Further reading ..
- :ref:`settings engine`
- :ref:`engine settings` & :ref:`engine file`
.. toctree::
:maxdepth: 3
settings
.. toctree::
:maxdepth: 1
private-engines
recoll
sql-engines
nosql-engines
search-indexer-engines
command-line-engines
searx.engines.xpath

View File

@ -1,5 +1,3 @@
.. _nosql engines:
=============== ===============
NoSQL databases NoSQL databases
=============== ===============
@ -10,16 +8,6 @@ NoSQL databases
- `redis.io <https://redis.io/>`_ - `redis.io <https://redis.io/>`_
- `MongoDB <https://www.mongodb.com>`_ - `MongoDB <https://www.mongodb.com>`_
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
The following `NoSQL databases`_ are supported: The following `NoSQL databases`_ are supported:
- :ref:`engine redis_server` - :ref:`engine redis_server`
@ -42,8 +30,15 @@ can still add them and limit the access by setting ``tokens`` as described in
section :ref:`private engines`. section :ref:`private engines`.
Configure the engines
=====================
`NoSQL databases`_ are used for storing arbitrary data without first defining
their structure.
Extra Dependencies Extra Dependencies
================== ------------------
For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to
install additional packages in Python's Virtual Environment of your SearXNG install additional packages in Python's Virtual Environment of your SearXNG
@ -54,13 +49,6 @@ instance. To switch into the environment (:ref:`searxng-src`) you can use
(searxng-pyenv)$ pip install ... (searxng-pyenv)$ pip install ...
Configure the engines
=====================
`NoSQL databases`_ are used for storing arbitrary data without first defining
their structure.
.. _engine redis_server: .. _engine redis_server:
Redis Server Redis Server
@ -74,9 +62,29 @@ Redis Server
- redis.io_ - redis.io_
- :origin:`redis_server.py <searx/engines/redis_server.py>` - :origin:`redis_server.py <searx/engines/redis_server.py>`
.. automodule:: searx.engines.redis_server
:members:
Redis is an open source (BSD licensed), in-memory data structure (key value
based) store. Before configuring the ``redis_server`` engine, you must install
the dependency redis_.
Select a database to search in and set its index in the option ``db``. You can
either look for exact matches or use partial keywords to find what you are
looking for by configuring ``exact_match_only``. You find an example
configuration below:
.. code:: yaml
# Required dependency: redis
- name: myredis
shortcut : rds
engine: redis_server
exact_match_only: false
host: '127.0.0.1'
port: 6379
enable_http: true
password: ''
db: 0
.. _engine mongodb: .. _engine mongodb:
@ -91,7 +99,37 @@ MongoDB
- MongoDB_ - MongoDB_
- :origin:`mongodb.py <searx/engines/mongodb.py>` - :origin:`mongodb.py <searx/engines/mongodb.py>`
MongoDB_ is a document based database program that handles JSON like data.
Before configuring the ``mongodb`` engine, you must install the dependency
redis_.
.. automodule:: searx.engines.mongodb In order to query MongoDB_, you have to select a ``database`` and a
:members: ``collection``. Furthermore, you have to select a ``key`` that is going to be
searched. MongoDB_ also supports the option ``exact_match_only``, so configure
it as you wish. Below is an example configuration for using a MongoDB
collection:
.. code:: yaml
# MongoDB engine
# Required dependency: pymongo
- name: mymongo
engine: mongodb
shortcut: md
exact_match_only: false
host: '127.0.0.1'
port: 27017
enable_http: true
results_per_page: 20
database: 'business'
collection: 'reviews' # name of the db collection
key: 'name' # key in the collection to search for
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View File

@ -0,0 +1,49 @@
.. _private engines:
============================
Private Engines (``tokens``)
============================
Administrators might find themselves wanting to limit access to some of the
enabled engines on their instances. It might be because they do not want to
expose some private information through :ref:`offline engines`. Or they would
rather share engines only with their trusted friends or colleagues.
To solve this issue the concept of *private engines* exists.
A new option was added to engines named `tokens`. It expects a list of
strings. If the user making a request presents one of the tokens of an engine,
they can access information about the engine and make search requests.
Example configuration to restrict access to the Arch Linux Wiki engine:
.. code:: yaml
- name: arch linux wiki
engine: archlinux
shortcut: al
tokens: [ 'my-secret-token' ]
Unless a user has configured the right token, the engine is going
to be hidden from him/her. It is not going to be included in the
list of engines on the Preferences page and in the output of
`/config` REST API call.
Tokens can be added to one's configuration on the Preferences page
under "Engine tokens". The input expects a comma separated list of
strings.
The distribution of the tokens from the administrator to the users
is not carved in stone. As providing access to such engines
implies that the admin knows and trusts the user, we do not see
necessary to come up with a strict process. Instead,
we would like to add guidelines to the documentation of the feature.
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View File

@ -0,0 +1,50 @@
.. _engine recoll:
=============
Recoll Engine
=============
.. sidebar:: info
- `Recoll <https://www.lesbonscomptes.com/recoll/>`_
- `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_
- :origin:`searx/engines/recoll.py`
Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_
does not offer WEB or API access, this can be achieved using recoll-webui_
Configuration
=============
You must configure the following settings:
``base_url``:
Location where recoll-webui can be reached.
``mount_prefix``:
Location where the file hierarchy is mounted on your *local* filesystem.
``dl_prefix``:
Location where the file hierarchy as indexed by recoll can be reached.
``search_dir``:
Part of the indexed file hierarchy to be search, if empty the full domain is
searched.
Example
=======
Scenario:
#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``,
#. the Recoll search interface can be reached at https://recoll.example.org/ and
#. the contents of this filesystem can be reached though https://download.example.org/reference
.. code:: yaml
base_url: https://recoll.example.org/
mount_prefix: /export/documents
dl_prefix: https://download.example.org
search_dir: ''

View File

@ -0,0 +1,136 @@
====================
Local Search Engines
====================
.. sidebar:: further read
- `Comparison to alternatives
<https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
Administrators might find themselves wanting to integrate locally running search
engines. The following ones are supported for now:
* `Elasticsearch`_
* `Meilisearch`_
* `Solr`_
Each search engine is powerful, capable of full-text search. All of the engines
above are added to ``settings.yml`` just commented out, as you have to
``base_url`` for all them.
Please note that if you are not using HTTPS to access these engines, you have to enable
HTTP requests by setting ``enable_http`` to ``True``.
Furthermore, if you do not want to expose these engines on a public instance, you
can still add them and limit the access by setting ``tokens`` as described in
section :ref:`private engines`.
.. _engine meilisearch:
MeiliSearch
===========
.. sidebar:: info
- :origin:`meilisearch.py <searx/engines/meilisearch.py>`
- `MeiliSearch <https://www.meilisearch.com>`_
- `MeiliSearch Documentation <https://docs.meilisearch.com/>`_
- `Install MeiliSearch
<https://docs.meilisearch.com/learn/getting_started/installation.html>`_
MeiliSearch_ is aimed at individuals and small companies. It is designed for
small-scale (less than 10 million documents) data collections. E.g. it is great
for storing web pages you have visited and searching in the contents later.
The engine supports faceted search, so you can search in a subset of documents
of the collection. Furthermore, you can search in MeiliSearch_ instances that
require authentication by setting ``auth_token``.
Here is a simple example to query a Meilisearch instance:
.. code:: yaml
- name: meilisearch
engine: meilisearch
shortcut: mes
base_url: http://localhost:7700
index: my-index
enable_http: true
.. _engine elasticsearch:
Elasticsearch
=============
.. sidebar:: info
- :origin:`elasticsearch.py <searx/engines/elasticsearch.py>`
- `Elasticsearch <https://www.elastic.co/elasticsearch/>`_
- `Elasticsearch Guide
<https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_
- `Install Elasticsearch
<https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_
Elasticsearch_ supports numerous ways to query the data it is storing. At the
moment the engine supports the most popular search methods (``query_type``):
- ``match``,
- ``simple_query_string``,
- ``term`` and
- ``terms``.
If none of the methods fit your use case, you can select ``custom`` query type
and provide the JSON payload to submit to Elasticsearch in
``custom_query_json``.
The following is an example configuration for an Elasticsearch_ instance with
authentication configured to read from ``my-index`` index.
.. code:: yaml
- name: elasticsearch
shortcut: es
engine: elasticsearch
base_url: http://localhost:9200
username: elastic
password: changeme
index: my-index
query_type: match
# custom_query_json: '{ ... }'
enable_http: true
.. _engine solr:
Solr
====
.. sidebar:: info
- :origin:`solr.py <searx/engines/solr.py>`
- `Solr <https://solr.apache.org>`_
- `Solr Resources <https://solr.apache.org/resources.html>`_
- `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_
Solr_ is a popular search engine based on Lucene, just like Elasticsearch_. But
instead of searching in indices, you can search in collections.
This is an example configuration for searching in the collection
``my-collection`` and get the results in ascending order.
.. code:: yaml
- name: solr
engine: solr
shortcut: slr
base_url: http://localhost:8983
collection: my-collection
sort: asc
enable_http: true
Acknowledgment
==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View File

@ -4,11 +4,6 @@
XPath Engine XPath Engine
============ ============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.xpath .. automodule:: searx.engines.xpath
:members: :members:

View File

@ -0,0 +1,732 @@
.. _settings.yml:
================
``settings.yml``
================
This page describe the options possibilities of the :origin:`searx/settings.yml`
file.
.. sidebar:: Further reading ..
- :ref:`use_default_settings.yml`
- :ref:`search API`
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. _settings location:
settings.yml location
=====================
The initial ``settings.yml`` we be load from these locations:
1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
2. ``/etc/searxng/settings.yml``
If these files don't exist (or are empty or can't be read), SearXNG uses the
:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to
see how you can simplify your *user defined* ``settings.yml``.
.. _settings global:
Global Settings
===============
.. _settings brand:
``brand:``
----------
.. code:: yaml
brand:
issue_url: https://github.com/searxng/searxng/issues
docs_url: https://docs.searxng.org
public_instances: https://searx.space
wiki_url: https://github.com/searxng/searxng/wiki
``issue_url`` :
If you host your own issue tracker change this URL.
``docs_url`` :
If you host your own documentation change this URL.
``public_instances`` :
If you host your own https://searx.space change this URL.
``wiki_url`` :
Link to your wiki (or ``false``)
.. _settings general:
``general:``
------------
.. code:: yaml
general:
debug: false
instance_name: "SearXNG"
privacypolicy_url: false
donation_url: false
contact_url: false
enable_metrics: true
``debug`` : ``$SEARXNG_DEBUG``
Allow a more detailed log if you run SearXNG directly. Display *detailed* error
messages in the browser too, so this must be deactivated in production.
``donation_url`` :
Set value to ``true`` to use your own donation page written in the
:ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable
the donation link altogether.
``privacypolicy_url``:
Link to privacy policy.
``contact_url``:
Contact ``mailto:`` address or WEB form.
``enable_metrics``:
Enabled by default. Record various anonymous metrics availabled at ``/stats``,
``/stats/errors`` and ``/preferences``.
.. _settings search:
``search:``
-----------
.. code:: yaml
search:
safe_search: 0
autocomplete: ""
default_lang: ""
ban_time_on_fail: 5
max_ban_time_on_fail: 120
suspended_times:
SearxEngineAccessDenied: 86400
SearxEngineCaptcha: 86400
SearxEngineTooManyRequests: 3600
cf_SearxEngineCaptcha: 1296000
cf_SearxEngineAccessDenied: 86400
recaptcha_SearxEngineCaptcha: 604800
formats:
- html
``safe_search``:
Filter results.
- ``0``: None
- ``1``: Moderate
- ``2``: Strict
``autocomplete``:
Existing autocomplete backends, leave blank to turn it off.
- ``dbpedia``
- ``duckduckgo``
- ``google``
- ``startpage``
- ``swisscows``
- ``qwant``
- ``wikipedia``
``default_lang``:
Default search language - leave blank to detect from browser information or
use codes from :origin:`searx/languages.py`.
``languages``:
List of available languages - leave unset to use all codes from
:origin:`searx/languages.py`. Otherwise list codes of available languages.
The ``all`` value is shown as the ``Default language`` in the user interface
(in most cases, it is meant to send the query without a language parameter ;
in some cases, it means the English language) Example:
.. code:: yaml
languages:
- all
- en
- en-US
- de
- it-IT
- fr
- fr-BE
``ban_time_on_fail``:
Ban time in seconds after engine errors.
``max_ban_time_on_fail``:
Max ban time in seconds after engine errors.
``suspended_times``:
Engine suspension time after error (in seconds; set to 0 to disable)
``SearxEngineAccessDenied``: 86400
For error "Access denied" and "HTTP error [402, 403]"
``SearxEngineCaptcha``: 86400
For error "CAPTCHA"
``SearxEngineTooManyRequests``: 3600
For error "Too many request" and "HTTP error 429"
Cloudflare CAPTCHA:
- ``cf_SearxEngineCaptcha``: 1296000
- ``cf_SearxEngineAccessDenied``: 86400
Google CAPTCHA:
- ``recaptcha_SearxEngineCaptcha``: 604800
``formats``:
Result formats available from web, remove format to deny access (use lower
case).
- ``html``
- ``csv``
- ``json``
- ``rss``
.. _settings server:
``server:``
-----------
.. code:: yaml
server:
base_url: http://example.org/location # change this!
port: 8888
bind_address: "127.0.0.1"
secret_key: "ultrasecretkey" # change this!
limiter: false
image_proxy: false
default_http_headers:
X-Content-Type-Options : nosniff
X-XSS-Protection : 1; mode=block
X-Download-Options : noopen
X-Robots-Tag : noindex, nofollow
Referrer-Policy : no-referrer
``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>`
The base URL where SearXNG is deployed. Used to create correct inbound links.
If you change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>`
Port number and *bind address* of the SearXNG web application if you run it
directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG
services running behind a proxy and using socket communications. If you
change the value, don't forget to rebuild instance's environment
(:ref:`utils/brand.env <make buildenv>`)
``secret_key`` : ``$SEARXNG_SECRET``
Used for cryptography purpose.
.. _limiter:
``limiter`` :
Rate limit the number of request on the instance, block some bots. The
:ref:`limiter src` requires a :ref:`settings redis` database.
.. _image_proxy:
``image_proxy`` :
Allow your instance of SearXNG of being able to proxy images. Uses memory space.
.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
``default_http_headers`` :
Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
.. _settings ui:
``ui:``
-------
.. _cache busting:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
.. code:: yaml
ui:
static_use_hash: false
default_locale: ""
query_in_title: false
infinite_scroll: false
center_alignment: false
cache_url: https://web.archive.org/web/
default_theme: simple
theme_args:
simple_style: auto
.. _static_use_hash:
``static_use_hash`` :
Enables `cache busting`_ of static files.
``default_locale`` :
SearXNG interface language. If blank, the locale is detected by using the
browser language. If it doesn't work, or you are deploying a language
specific instance of searx, a locale can be defined using an ISO language
code, like ``fr``, ``en``, ``de``.
``query_in_title`` :
When true, the result page's titles contains the query it decreases the
privacy, since the browser can records the page titles.
``infinite_scroll``:
When true, automatically loads the next page when scrolling to bottom of the current page.
``center_alignment`` : default ``false``
When enabled, the results are centered instead of being in the left (or RTL)
side of the screen. This setting only affects the *desktop layout*
(:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
.. cache_url:
``cache_url`` : ``https://web.archive.org/web/``
URL prefix of the internet archive or cache, don't forgett trailing slash (if
needed). The default is https://web.archive.org/web/ alternatives are:
- https://webcache.googleusercontent.com/search?q=cache:
- https://archive.today/
``default_theme`` :
Name of the theme you want to use by default on your SearXNG instance.
``theme_args.simple_style``:
Style of simple theme: ``auto``, ``light``, ``dark``
``results_on_new_tab``:
Open result links in a new tab by default.
.. _settings redis:
``redis:``
----------
.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
will find a description to test your redis connection in SerXNG. When using
sockets, don't forget to check the access rights on the socket::
ls -la /usr/local/searxng-redis/run/redis.sock
srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
In this example read/write access is given to the *searxng-redis* group. To get
access rights to redis instance (the socket), your SearXNG (or even your
developer) account needs to be added to the *searxng-redis* group.
``url``
URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
redis://[[username]:[password]]@localhost:6379/0
rediss://[[username]:[password]]@localhost:6379/0
unix://[[username]:[password]]@/path/to/socket.sock?db=0
.. admonition:: Tip for developers
To set up a local redis instance, first set the socket path of the Redis DB
in your YAML setting:
.. code:: yaml
redis:
url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
Then use the following commands to install the redis instance ::
$ ./manage redis.build
$ sudo -H ./manage redis.install
$ sudo -H ./manage redis.addgrp "${USER}"
# don't forget to logout & login to get member of group
.. _settings outgoing:
``outgoing:``
-------------
Communication with search engines.
.. code:: yaml
outgoing:
request_timeout: 2.0 # default timeout in seconds, can be override by engine
max_request_timeout: 10.0 # the maximum timeout in seconds
useragent_suffix: "" # information like an email address to the administrator
pool_connections: 100 # Maximum number of allowable connections, or null
# for no limits. The default is 100.
pool_maxsize: 10 # Number of allowable keep-alive connections, or null
# to always allow. The default is 10.
enable_http2: true # See https://www.python-httpx.org/http2/
# uncomment below section if you want to use a custom server certificate
# see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
# and https://www.python-httpx.org/compatibility/#ssl-configuration
# verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
#
# uncomment below section if you want to use a proxyq see: SOCKS proxies
# https://2.python-requests.org/en/latest/user/advanced/#proxies
# are also supported: see
# https://2.python-requests.org/en/latest/user/advanced/#socks
#
# proxies:
# all://:
# - http://proxy1:8080
# - http://proxy2:8080
#
# using_tor_proxy: true
#
# Extra seconds to add in order to account for the time taken by the proxy
#
# extra_proxy_timeout: 10.0
#
``request_timeout`` :
Global timeout of the requests made to others engines in seconds. A bigger
timeout will allow to wait for answers from slow engines, but in consequence
will slow SearXNG reactivity (the result page may take the time specified in the
timeout to load). Can be override by :ref:`settings engine`
``useragent_suffix`` :
Suffix to the user-agent SearXNG uses to send requests to others engines. If an
engine wish to block you, a contact info here may be useful to avoid that.
``keepalive_expiry`` :
Number of seconds to keep a connection in the pool. By default 5.0 seconds.
.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
``proxies`` :
Define one or more proxies you wish to use, see `httpx proxies`_.
If there are more than one proxy for one protocol (http, https),
requests to the engines are distributed in a round-robin fashion.
``source_ips`` :
If you use multiple network interfaces, define from which IP the requests must
be made. Example:
* ``0.0.0.0`` any local IPv4 address.
* ``::`` any local IPv6 address.
* ``192.168.0.1``
* ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
* ``fe80::60a2:1691:e5a2:ee1f``
* ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
* ``[ 192.168.0.1, fe80::/126 ]``
``retries`` :
Number of retry in case of an HTTP error. On each retry, SearXNG uses an
different proxy and source ip.
``retry_on_http_error`` :
Retry request on some HTTP status code.
Example:
* ``true`` : on HTTP status code between 400 and 599.
* ``403`` : on HTTP status code 403.
* ``[403, 429]``: on HTTP status code 403 and 429.
``enable_http2`` :
Enable by default. Set to ``false`` to disable HTTP/2.
.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
Allow to specify a path to certificate.
see `httpx verification defaults`_.
In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
see `httpx ssl configuration`_.
``max_redirects`` :
30 by default. Maximum redirect before it is an error.
.. _settings categories_as_tabs:
``categories_as_tabs:``
-----------------------
A list of the categories that are displayed as tabs in the user interface.
Categories not listed here can still be searched with the :ref:`search-syntax`.
.. code-block:: yaml
categories_as_tabs:
general:
images:
videos:
news:
map:
music:
it:
science:
files:
social media:
Engines are added to ``categories:`` (compare :ref:`engine categories`), the
categories listed in ``categories_as_tabs`` are shown as tabs in the UI. If
there are no active engines in a category, the tab is not displayed (e.g. if a
user disables all engines in a category).
On the preferences page (``/preferences``) -- under *engines* -- there is an
additional tab, called *other*. In this tab are all engines listed that are not
in one of the UI tabs (not included in ``categories_as_tabs``).
.. _settings engine:
Engine settings
===============
.. sidebar:: Further reading ..
- :ref:`configured engines`
- :ref:`engines-dev`
In the code example below a *full fledged* example of a YAML setup from a dummy
engine is shown. Most of the options have a default value or even are optional.
.. code:: yaml
- name: example engine
engine: example
shortcut: demo
base_url: 'https://{language}.example.com/'
send_accept_language_header: false
categories: general
timeout: 3.0
api_key: 'apikey'
disabled: false
language: en_US
tokens: [ 'my-secret-token' ]
weight: 1
display_error_messages: true
about:
website: https://example.com
wikidata_id: Q306656
official_api_documentation: https://example.com/api-doc
use_official_api: true
require_api_key: true
results: HTML
enable_http: false
enable_http2: false
retries: 1
retry_on_http_error: true # or 403 or [404, 429]
max_connections: 100
max_keepalive_connections: 10
keepalive_expiry: 5.0
proxies:
http:
- http://proxy1:8080
- http://proxy2:8080
https:
- http://proxy1:8080
- http://proxy2:8080
- socks5://user:password@proxy3:1080
- socks5h://user:password@proxy4:1080
``name`` :
Name that will be used across SearXNG to define this engine. In settings, on
the result page...
``engine`` :
Name of the python file used to handle requests and responses to and from this
search engine.
``shortcut`` :
Code used to execute bang requests (in this case using ``!bi``)
``base_url`` : optional
Part of the URL that should be stable across every request. Can be useful to
use multiple sites using only one engine, or updating the site URL without
touching at the code.
``send_accept_language_header`` :
Several engines that support languages (or regions) deal with the HTTP header
``Accept-Language`` to build a response that fits to the locale. When this
option is activated, the language (locale) that is selected by the user is used
to build and send a ``Accept-Language`` header in the request to the origin
search engine.
.. _engine categories:
``categories`` : optional
Specifies to which categories the engine should be added. Engines can be
assigned to multiple categories.
Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the
UI. A search in a tab (in the UI) will query all engines that are active in
this tab. In the preferences page (``/preferences``) -- under *engines* --
users can select what engine should be active when querying in this tab.
Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines
in a category, regardless of whether they are active or not, or whether they
are in a tab of the UI or not. For example, ``!dictionaries`` can be used to
query all search engines in that category (group).
``timeout`` : optional
Timeout of the search with the current search engine. **Be careful, it will
modify the global timeout of SearXNG.**
``api_key`` : optional
In a few cases, using an API needs the use of a secret key. How to obtain them
is described in the file.
``disabled`` : optional
To disable by default the engine, but not deleting it. It will allow the user
to manually activate it in the settings.
``inactive``: optional
Remove the engine from the settings (*disabled & removed*).
``language`` : optional
If you want to use another language for a specific engine, you can define it
by using the ISO code of language (and region), like ``fr``, ``en-US``,
``de-DE``.
``tokens`` : optional
A list of secret tokens to make this engine *private*, more details see
:ref:`private engines`.
``weight`` : default ``1``
Weighting of the results of this engine.
``display_error_messages`` : default ``true``
When an engine returns an error, the message is displayed on the user interface.
``network`` : optional
Use the network configuration from another engine.
In addition, there are two default networks:
- ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
- ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
.. note::
A few more options are possible, but they are pretty specific to some
engines, and so won't be described here.
Example: Multilingual Search
----------------------------
SearXNG does not support true multilingual search. You have to use the language
prefix in your search query when searching in a different language.
But there is a workaround: By adding a new search engine with a different
language, SearXNG will search in your default and other language.
Example configuration in settings.yml for a German and English speaker:
.. code-block:: yaml
search:
default_lang : "de"
...
engines:
- name : google english
engine : google
language : en
...
When searching, the default google engine will return German results and
"google english" will return English results.
.. _settings use_default_settings:
use_default_settings
====================
.. sidebar:: ``use_default_settings: true``
- :ref:`settings location`
- :ref:`use_default_settings.yml`
- :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
The user defined ``settings.yml`` is loaded from the :ref:`settings location`
and can relied on the default configuration :origin:`searx/settings.yml` using:
``use_default_settings: true``
``server:``
In the following example, the actual settings are the default settings defined
in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
the ``bind_address``:
.. code-block:: yaml
use_default_settings: true
server:
secret_key: "ultrasecretkey" # change this!
bind_address: "0.0.0.0"
``engines:``
With ``use_default_settings: true``, each settings can be override in a
similar way, the ``engines`` section is merged according to the engine
``name``. In this example, SearXNG will load all the default engines, will
enable the ``bing`` engine and define a :ref:`token <private engines>` for
the arch linux engine:
.. code-block:: yaml
use_default_settings: true
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
- name: bing
disabled: false
``engines:`` / ``remove:``
It is possible to remove some engines from the default settings. The following
example is similar to the above one, but SearXNG doesn't load the the google
engine:
.. code-block:: yaml
use_default_settings:
engines:
remove:
- google
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
``engines:`` / ``keep_only:``
As an alternative, it is possible to specify the engines to keep. In the
following example, SearXNG has only two engines:
.. code-block:: yaml
use_default_settings:
engines:
keep_only:
- google
- duckduckgo
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: google
tokens: ['$ecretValue']
- name: duckduckgo
tokens: ['$ecretValue']

View File

@ -10,22 +10,12 @@ SQL Engines
- `PostgreSQL <https://www.postgresql.org>`_ - `PostgreSQL <https://www.postgresql.org>`_
- `MySQL <https://www.mysql.com>`_ - `MySQL <https://www.mysql.com>`_
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
With the *SQL engines* you can bind SQL databases into SearXNG. The following With the *SQL engines* you can bind SQL databases into SearXNG. The following
Relational Database Management System (RDBMS) are supported: Relational Database Management System (RDBMS) are supported:
- :ref:`engine sqlite` - :ref:`engine sqlite`
- :ref:`engine postgresql` - :ref:`engine postgresql`
- :ref:`engine mysql_server` & :ref:`engine mariadb_server` - :ref:`engine mysql_server`
All of the engines above are just commented out in the :origin:`settings.yml All of the engines above are just commented out in the :origin:`settings.yml
<searx/settings.yml>`, as you have to set the required attributes for the <searx/settings.yml>`, as you have to set the required attributes for the
@ -52,18 +42,6 @@ add them and limit the access by setting ``tokens`` as described in section
:ref:`private engines`. :ref:`private engines`.
Extra Dependencies
==================
For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
install additional packages in Python's Virtual Environment of your SearXNG
instance. To switch into the environment (:ref:`searxng-src`) you can use
:ref:`searxng.sh`::
$ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install ...
Configure the engines Configure the engines
===================== =====================
@ -86,8 +64,45 @@ SQLite
- :origin:`sqlite.py <searx/engines/sqlite.py>` - :origin:`sqlite.py <searx/engines/sqlite.py>`
.. automodule:: searx.engines.sqlite .. _MediathekView: https://mediathekview.de/
:members:
SQLite is a small, fast and reliable SQL database engine. It does not require
any extra dependency. To demonstrate the power of database engines, here is a
more complex example which reads from a MediathekView_ (DE) movie database. For
this example of the SQlite engine download the database:
- https://liste.mediathekview.de/filmliste-v2.db.bz2
and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g
Query to test: ``!mediathekview concert``
.. code:: yaml
- name: mediathekview
engine: sqlite
disabled: False
categories: general
result_template: default.html
database: searx/data/filmliste-v2.db
query_str: >-
SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
description AS content
FROM film
WHERE title LIKE :wildcard OR description LIKE :wildcard
ORDER BY duration DESC
Extra Dependencies
------------------
For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to
install additional packages in Python's Virtual Environment of your SearXNG
instance. To switch into the environment (:ref:`searxng-src`) you can use
:ref:`searxng.sh`::
$ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install ...
.. _engine postgresql: .. _engine postgresql:
@ -100,10 +115,20 @@ PostgreSQL
.. sidebar:: info .. sidebar:: info
- :origin:`postgresql.py <searx/engines/postgresql.py>` - :origin:`postgresql.py <searx/engines/postgresql.py>`
- ``pip install`` `psycopg2-binary <psycopg2>`_ - ``pip install`` psycopg2_
.. automodule:: searx.engines.postgresql PostgreSQL is a powerful and robust open source database. Before configuring
:members: the PostgreSQL engine, you must install the dependency ``psychopg2``. You can
find an example configuration below:
.. code:: yaml
- name: my_database
engine: postgresql
database: my_database
username: searxng
password: password
query_str: 'SELECT * from my_table WHERE my_column = %(query)s'
.. _engine mysql_server: .. _engine mysql_server:
@ -115,20 +140,27 @@ MySQL
- :origin:`mysql_server.py <searx/engines/mysql_server.py>` - :origin:`mysql_server.py <searx/engines/mysql_server.py>`
- ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>` - ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>`
MySQL is said to be the most popular open source database. Before enabling MySQL
engine, you must install the package ``mysql-connector-python``.
.. automodule:: searx.engines.mysql_server The authentication plugin is configurable by setting ``auth_plugin`` in the
:members: attributes. By default it is set to ``caching_sha2_password``. This is an
example configuration for querying a MySQL server:
.. _engine mariadb_server: .. code:: yaml
MariaDB - name: my_database
-------- engine: mysql_server
database: my_database
.. sidebar:: info username: searxng
password: password
- :origin:`mariadb_server.py <searx/engines/mariadb_server.py>` limit: 5
- ``pip install`` :pypi:`mariadb <mariadb>` query_str: 'SELECT * from my_table WHERE my_column=%(query)s'
.. automodule:: searx.engines.mariadb_server Acknowledgment
:members: ==============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.

View File

@ -4,8 +4,8 @@ Administrator documentation
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Contents
settings/index
installation installation
installation-docker installation-docker
installation-scripts installation-scripts
@ -15,8 +15,7 @@ Administrator documentation
installation-apache installation-apache
update-searxng update-searxng
answer-captcha answer-captcha
searx.favicons engines/index
searx.limiter
api api
architecture architecture
plugins plugins

View File

@ -61,7 +61,7 @@ section might give you some guidance.
- `Apache Fedora`_ - `Apache Fedora`_
- `Apache directives`_ - `Apache directives`_
.. contents:: .. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
@ -190,7 +190,7 @@ Debian:
Apache modules Apache modules
-------------- --------------
To load additional modules, in most distributions you have to uncomment the To load additional modules, in most distributions you have to un-comment the
lines with the corresponding LoadModule_ directive, except in :ref:`Debian's lines with the corresponding LoadModule_ directive, except in :ref:`Debian's
Apache layout`. Apache layout`.

View File

@ -43,7 +43,7 @@ of this container:
- enables :ref:`limiter <limiter>` to protect against bots - enables :ref:`limiter <limiter>` to protect against bots
- enables :ref:`image proxy <image_proxy>` for better privacy - enables :ref:`image proxy <image_proxy>` for better privacy
- enables :ref:`cache busting <static_use_hash>` to save bandwidth - enables :ref:`cache busting <static_use_hash>` to save bandwith
---- ----
@ -92,9 +92,6 @@ instance using `docker run <https://docs.docker.com/engine/reference/run/>`_:
searxng/searxng searxng/searxng
2f998.... # container's ID 2f998.... # container's ID
The environment variables UWSGI_WORKERS and UWSGI_THREADS overwrite the default
number of UWSGI processes and UWSGI threads specified in `/etc/searxng/uwsgi.ini`.
Open your WEB browser and visit the URL: Open your WEB browser and visit the URL:
.. code:: sh .. code:: sh
@ -111,7 +108,7 @@ can modify these files according to your needs and restart the Docker image.
Use command ``container ls`` to list running containers, add flag `-a Use command ``container ls`` to list running containers, add flag `-a
<https://docs.docker.com/engine/reference/commandline/container_ls>`__ to list <https://docs.docker.com/engine/reference/commandline/container_ls>`__ to list
exited containers also. With ``container stop`` a running container can be exited containers also. With ``container stop`` a running container can be
stopped. To get rid of a container use ``container rm``: stoped. To get rid of a container use ``container rm``:
.. code:: sh .. code:: sh

View File

@ -41,7 +41,7 @@ section might give you some guidance.
- `uWSGI support from nginx`_ - `uWSGI support from nginx`_
.. contents:: .. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View File

@ -4,7 +4,7 @@
Step by step installation Step by step installation
========================= =========================
.. contents:: .. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
@ -73,7 +73,7 @@ Configuration
.. sidebar:: ``use_default_settings: True`` .. sidebar:: ``use_default_settings: True``
- :ref:`settings.yml` - :ref:`settings global`
- :ref:`settings location` - :ref:`settings location`
- :ref:`settings use_default_settings` - :ref:`settings use_default_settings`
- :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>` - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
@ -86,7 +86,7 @@ below. This setup:
- enables :ref:`limiter <limiter>` to protect against bots - enables :ref:`limiter <limiter>` to protect against bots
- enables :ref:`image proxy <image_proxy>` for better privacy - enables :ref:`image proxy <image_proxy>` for better privacy
- enables :ref:`cache busting <static_use_hash>` to save bandwidth - enables :ref:`cache busting <static_use_hash>` to save bandwith
Modify the ``/etc/searxng/settings.yml`` to your needs: Modify the ``/etc/searxng/settings.yml`` to your needs:
@ -96,7 +96,7 @@ Modify the ``/etc/searxng/settings.yml`` to your needs:
.. literalinclude:: ../../utils/templates/etc/searxng/settings.yml .. literalinclude:: ../../utils/templates/etc/searxng/settings.yml
:language: yaml :language: yaml
:end-before: # hostnames: :end-before: # hostname_replace:
To see the entire file jump to :origin:`utils/templates/etc/searxng/settings.yml` To see the entire file jump to :origin:`utils/templates/etc/searxng/settings.yml`
@ -104,7 +104,7 @@ Modify the ``/etc/searxng/settings.yml`` to your needs:
.. literalinclude:: ../../searx/settings.yml .. literalinclude:: ../../searx/settings.yml
:language: yaml :language: yaml
:end-before: # hostnames: :end-before: # hostname_replace:
To see the entire file jump to :origin:`searx/settings.yml` To see the entire file jump to :origin:`searx/settings.yml`

View File

@ -9,7 +9,7 @@ uWSGI
- `systemd.unit`_ - `systemd.unit`_
- `uWSGI Emperor`_ - `uWSGI Emperor`_
.. contents:: .. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry

View File

@ -1,251 +0,0 @@
.. _favicons:
========
Favicons
========
.. sidebar:: warning
Don't activate the favicons before reading the documentation.
.. contents::
:depth: 2
:local:
:backlinks: entry
Activating the favicons in SearXNG is very easy, but this **generates a
significantly higher load** in the client/server communication and increases
resources needed on the server.
To mitigate these disadvantages, various methods have been implemented,
including a *cache*. The cache must be parameterized according to your own
requirements and maintained regularly.
To activate favicons in SearXNG's result list, set a default
``favicon_resolver`` in the :ref:`search <settings search>` settings:
.. code:: yaml
search:
favicon_resolver: "duckduckgo"
By default and without any extensions, SearXNG serves these resolvers:
- ``duckduckgo``
- ``allesedv``
- ``google``
- ``yandex``
With the above setting favicons are displayed, the user has the option to
deactivate this feature in his settings. If the user is to have the option of
selecting from several *resolvers*, a further setting is required / but this
setting will be discussed :ref:`later <register resolvers>` in this article,
first we have to setup the favicons cache.
Infrastructure
==============
The infrastructure for providing the favicons essentially consists of three
parts:
- :py:obj:`Favicons-Proxy <.favicons.proxy>` (aka *proxy*)
- :py:obj:`Favicons-Resolvers <.favicons.resolvers>` (aka *resolver*)
- :py:obj:`Favicons-Cache <.favicons.cache>` (aka *cache*)
To protect the privacy of users, the favicons are provided via a *proxy*. This
*proxy* is automatically activated with the above activation of a *resolver*.
Additional requests are required to provide the favicons: firstly, the *proxy*
must process the incoming requests and secondly, the *resolver* must make
outgoing requests to obtain the favicons from external sources.
A *cache* has been developed to massively reduce both, incoming and outgoing
requests. This *cache* is also activated automatically with the above
activation of a *resolver*. In its defaults, however, the *cache* is minimal
and not well suitable for a production environment!
.. _favicon cache setup:
Setting up the cache
====================
To parameterize the *cache* and more settings of the favicons infrastructure, a
TOML_ configuration is created in the file ``/etc/searxng/favicons.toml``.
.. code:: toml
[favicons]
cfg_schema = 1 # config's schema version no.
[favicons.cache]
db_url = "/var/cache/searxng/faviconcache.db" # default: "/tmp/faviconcache.db"
LIMIT_TOTAL_BYTES = 2147483648 # 2 GB / default: 50 MB
# HOLD_TIME = 5184000 # 60 days / default: 30 days
# BLOB_MAX_BYTES = 40960 # 40 KB / default 20 KB
# MAINTENANCE_MODE = "off" # default: "auto"
# MAINTENANCE_PERIOD = 600 # 10min / default: 1h
:py:obj:`cfg_schema <.FaviconConfig.cfg_schema>`:
Is required to trigger any processes required for future upgrades / don't
change it.
:py:obj:`cache.db_url <.FaviconCacheConfig.db_url>`:
The path to the (SQLite_) database file. The default path is in the `/tmp`_
folder, which is deleted on every reboot and is therefore unsuitable for a
production environment. The FHS_ provides the folder for the
application cache
The FHS_ provides the folder `/var/cache`_ for the cache of applications, so a
suitable storage location of SearXNG's caches is folder ``/var/cache/searxng``.
In container systems, a volume should be mounted for this folder and in a
standard installation (compare :ref:`create searxng user`), the folder must be
created and the user under which the SearXNG process is running must be given
write permission to this folder.
.. code:: bash
$ sudo mkdir /var/cache/searxng
$ sudo chown root:searxng /var/cache/searxng/
$ sudo chmod g+w /var/cache/searxng/
:py:obj:`cache.LIMIT_TOTAL_BYTES <.FaviconCacheConfig.LIMIT_TOTAL_BYTES>`:
Maximum of bytes stored in the cache of all blobs. The limit is only reached
at each maintenance interval after which the oldest BLOBs are deleted; the
limit is exceeded during the maintenance period.
.. attention::
If the maintenance period is too long or maintenance is switched
off completely, the cache grows uncontrollably.
SearXNG hosters can change other parameters of the cache as required:
- :py:obj:`cache.HOLD_TIME <.FaviconCacheConfig.HOLD_TIME>`
- :py:obj:`cache.BLOB_MAX_BYTES <.FaviconCacheConfig.BLOB_MAX_BYTES>`
Maintenance of the cache
------------------------
Regular maintenance of the cache is required! By default, regular maintenance
is triggered automatically as part of the client requests:
- :py:obj:`cache.MAINTENANCE_MODE <.FaviconCacheConfig.MAINTENANCE_MODE>` (default ``auto``)
- :py:obj:`cache.MAINTENANCE_PERIOD <.FaviconCacheConfig.MAINTENANCE_PERIOD>` (default ``6000`` / 1h)
As an alternative to maintenance as part of the client request process, it is
also possible to carry out maintenance using an external process. For example,
by creating a :man:`crontab` entry for maintenance:
.. code:: bash
$ python -m searx.favicons cache maintenance
The following command can be used to display the state of the cache:
.. code:: bash
$ python -m searx.favicons cache state
.. _favicon proxy setup:
Proxy configuration
===================
Most of the options of the :py:obj:`Favicons-Proxy <.favicons.proxy>` are
already set sensibly with settings from the :ref:`settings.yml <searxng
settings.yml>` and should not normally be adjusted.
.. code:: toml
[favicons.proxy]
max_age = 5184000 # 60 days / default: 7 days (604800 sec)
:py:obj:`max_age <.FaviconProxyConfig.max_age>`:
The `HTTP Cache-Control max-age`_ response directive indicates that the
response remains fresh until N seconds after the response is generated. This
setting therefore determines how long a favicon remains in the client's cache.
As a rule, in the favicons infrastructure of SearXNG's this setting only
affects favicons whose byte size exceeds :ref:`BLOB_MAX_BYTES <favicon cache
setup>` (the other favicons that are already in the cache are embedded as
`data URL`_ in the :py:obj:`generated HTML <.favicons.proxy.favicon_url>`,
which can greatly reduce the number of additional requests).
.. _register resolvers:
Register resolvers
------------------
A :py:obj:`resolver <.favicon.resolvers>` is a function that obtains the favicon
from an external source. The resolver functions available to the user are
registered with their fully qualified name (FQN_) in a ``resolver_map``.
If no ``resolver_map`` is defined in the ``favicon.toml``, the favicon
infrastructure of SearXNG generates this ``resolver_map`` automatically
depending on the ``settings.yml``. SearXNG would automatically generate the
following TOML configuration from the following YAML configuration:
.. code:: yaml
search:
favicon_resolver: "duckduckgo"
.. code:: toml
[favicons.proxy.resolver_map]
"duckduckgo" = "searx.favicons.resolvers.duckduckgo"
If this automatism is not desired, then (and only then) a separate
``resolver_map`` must be created. For example, to give the user two resolvers to
choose from, the following configuration could be used:
.. code:: toml
[favicons.proxy.resolver_map]
"duckduckgo" = "searx.favicons.resolvers.duckduckgo"
"allesedv" = "searx.favicons.resolvers.allesedv"
# "google" = "searx.favicons.resolvers.google"
# "yandex" = "searx.favicons.resolvers.yandex"
.. note::
With each resolver, the resource requirement increases significantly.
The number of resolvers increases:
- the number of incoming/outgoing requests and
- the number of favicons to be stored in the cache.
In the following we list the resolvers available in the core of SearXNG, but via
the FQN_ it is also possible to implement your own resolvers and integrate them
into the *proxy*:
- :py:obj:`searx.favicons.resolvers.duckduckgo`
- :py:obj:`searx.favicons.resolvers.allesedv`
- :py:obj:`searx.favicons.resolvers.google`
- :py:obj:`searx.favicons.resolvers.yandex`
.. _SQLite:
https://www.sqlite.org/
.. _FHS:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
.. _`/var/cache`:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s05.html
.. _`/tmp`:
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html
.. _TOML:
https://toml.io/en/
.. _HTTP Cache-Control max-age:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#response_directives
.. _data URL:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs
.. _FQN: https://en.wikipedia.org/wiki/Fully_qualified_name

View File

@ -1,17 +0,0 @@
.. _limiter:
=======
Limiter
=======
.. sidebar:: info
The limiter requires a :ref:`Redis <settings redis>` database.
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.limiter
:members:

View File

@ -1,27 +0,0 @@
.. _searxng settings.yml:
========
Settings
========
.. sidebar:: Further reading ..
- :ref:`engine settings`
- :ref:`engine file`
.. toctree::
:maxdepth: 2
settings
settings_engine
settings_brand
settings_general
settings_search
settings_server
settings_ui
settings_redis
settings_outgoing
settings_categories_as_tabs

View File

@ -1,117 +0,0 @@
.. _settings.yml:
================
``settings.yml``
================
This page describe the options possibilities of the :origin:`searx/settings.yml`
file.
.. sidebar:: Further reading ..
- :ref:`use_default_settings.yml`
- :ref:`search API`
.. contents::
:depth: 2
:local:
:backlinks: entry
.. _settings location:
settings.yml location
=====================
The initial ``settings.yml`` we be load from these locations:
1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable.
2. ``/etc/searxng/settings.yml``
If these files don't exist (or are empty or can't be read), SearXNG uses the
:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to
see how you can simplify your *user defined* ``settings.yml``.
.. _settings use_default_settings:
use_default_settings
====================
.. sidebar:: ``use_default_settings: true``
- :ref:`settings location`
- :ref:`use_default_settings.yml`
- :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>`
The user defined ``settings.yml`` is loaded from the :ref:`settings location`
and can relied on the default configuration :origin:`searx/settings.yml` using:
``use_default_settings: true``
``server:``
In the following example, the actual settings are the default settings defined
in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and
the ``bind_address``:
.. code:: yaml
use_default_settings: true
server:
secret_key: "ultrasecretkey" # change this!
bind_address: "0.0.0.0"
``engines:``
With ``use_default_settings: true``, each settings can be override in a
similar way, the ``engines`` section is merged according to the engine
``name``. In this example, SearXNG will load all the default engines, will
enable the ``bing`` engine and define a :ref:`token <private engines>` for
the arch linux engine:
.. code:: yaml
use_default_settings: true
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
- name: bing
disabled: false
``engines:`` / ``remove:``
It is possible to remove some engines from the default settings. The following
example is similar to the above one, but SearXNG doesn't load the the google
engine:
.. code:: yaml
use_default_settings:
engines:
remove:
- google
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: arch linux wiki
tokens: ['$ecretValue']
``engines:`` / ``keep_only:``
As an alternative, it is possible to specify the engines to keep. In the
following example, SearXNG has only two engines:
.. code:: yaml
use_default_settings:
engines:
keep_only:
- google
- duckduckgo
server:
secret_key: "ultrasecretkey" # change this!
engines:
- name: google
tokens: ['$ecretValue']
- name: duckduckgo
tokens: ['$ecretValue']

View File

@ -1,25 +0,0 @@
.. _settings brand:
==========
``brand:``
==========
.. code:: yaml
brand:
issue_url: https://github.com/searxng/searxng/issues
docs_url: https://docs.searxng.org
public_instances: https://searx.space
wiki_url: https://github.com/searxng/searxng/wiki
``issue_url`` :
If you host your own issue tracker change this URL.
``docs_url`` :
If you host your own documentation change this URL.
``public_instances`` :
If you host your own https://searx.space change this URL.
``wiki_url`` :
Link to your wiki (or ``false``)

View File

@ -1,31 +0,0 @@
.. _settings categories_as_tabs:
=======================
``categories_as_tabs:``
=======================
A list of the categories that are displayed as tabs in the user interface.
Categories not listed here can still be searched with the :ref:`search-syntax`.
.. code:: yaml
categories_as_tabs:
general:
images:
videos:
news:
map:
music:
it:
science:
files:
social media:
Engines are added to ``categories:`` (compare :ref:`engine categories`), the
categories listed in ``categories_as_tabs`` are shown as tabs in the UI. If
there are no active engines in a category, the tab is not displayed (e.g. if a
user disables all engines in a category).
On the preferences page (``/preferences``) -- under *engines* -- there is an
additional tab, called *other*. In this tab are all engines listed that are not
in one of the UI tabs (not included in ``categories_as_tabs``).

View File

@ -1,244 +0,0 @@
.. _settings engine:
===========
``engine:``
===========
.. sidebar:: Further reading ..
- :ref:`configured engines`
- :ref:`engines-dev`
In the code example below a *full fledged* example of a YAML setup from a dummy
engine is shown. Most of the options have a default value or even are optional.
.. hint::
A few more options are possible, but they are pretty specific to some
engines (:ref:`engine implementations`).
.. code:: yaml
- name: example engine
engine: example
shortcut: demo
base_url: 'https://{language}.example.com/'
send_accept_language_header: false
categories: general
timeout: 3.0
api_key: 'apikey'
disabled: false
language: en_US
tokens: [ 'my-secret-token' ]
weight: 1
display_error_messages: true
about:
website: https://example.com
wikidata_id: Q306656
official_api_documentation: https://example.com/api-doc
use_official_api: true
require_api_key: true
results: HTML
# overwrite values from section 'outgoing:'
enable_http2: false
retries: 1
max_connections: 100
max_keepalive_connections: 10
keepalive_expiry: 5.0
using_tor_proxy: false
proxies:
http:
- http://proxy1:8080
- http://proxy2:8080
https:
- http://proxy1:8080
- http://proxy2:8080
- socks5://user:password@proxy3:1080
- socks5h://user:password@proxy4:1080
# other network settings
enable_http: false
retry_on_http_error: true # or 403 or [404, 429]
``name`` :
Name that will be used across SearXNG to define this engine. In settings, on
the result page...
``engine`` :
Name of the python file used to handle requests and responses to and from this
search engine.
``shortcut`` :
Code used to execute bang requests (in this case using ``!bi``)
``base_url`` : optional
Part of the URL that should be stable across every request. Can be useful to
use multiple sites using only one engine, or updating the site URL without
touching at the code.
``send_accept_language_header`` :
Several engines that support languages (or regions) deal with the HTTP header
``Accept-Language`` to build a response that fits to the locale. When this
option is activated, the language (locale) that is selected by the user is used
to build and send a ``Accept-Language`` header in the request to the origin
search engine.
.. _engine categories:
``categories`` : optional
Specifies to which categories the engine should be added. Engines can be
assigned to multiple categories.
Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the
UI. A search in a tab (in the UI) will query all engines that are active in
this tab. In the preferences page (``/preferences``) -- under *engines* --
users can select what engine should be active when querying in this tab.
Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines
in a category, regardless of whether they are active or not, or whether they
are in a tab of the UI or not. For example, ``!dictionaries`` can be used to
query all search engines in that category (group).
``timeout`` : optional
Timeout of the search with the current search engine. Overwrites
``request_timeout`` from :ref:`settings outgoing`. **Be careful, it will
modify the global timeout of SearXNG.**
``api_key`` : optional
In a few cases, using an API needs the use of a secret key. How to obtain them
is described in the file.
``disabled`` : optional
To disable by default the engine, but not deleting it. It will allow the user
to manually activate it in the settings.
``inactive``: optional
Remove the engine from the settings (*disabled & removed*).
``language`` : optional
If you want to use another language for a specific engine, you can define it
by using the ISO code of language (and region), like ``fr``, ``en-US``,
``de-DE``.
``tokens`` : optional
A list of secret tokens to make this engine *private*, more details see
:ref:`private engines`.
``weight`` : default ``1``
Weighting of the results of this engine.
``display_error_messages`` : default ``true``
When an engine returns an error, the message is displayed on the user interface.
``network`` : optional
Use the network configuration from another engine.
In addition, there are two default networks:
- ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)
- ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)
``enable_http`` : optional
Enable HTTP for this engine (by default only HTTPS is enabled).
``retry_on_http_error`` : optional
Retry request on some HTTP status code.
Example:
* ``true`` : on HTTP status code between 400 and 599.
* ``403`` : on HTTP status code 403.
* ``[403, 429]``: on HTTP status code 403 and 429.
``proxies`` :
Overwrites proxy settings from :ref:`settings outgoing`.
``using_tor_proxy`` :
Using tor proxy (``true``) or not (``false``) for this engine. The default is
taken from ``using_tor_proxy`` of the :ref:`settings outgoing`.
.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
``max_keepalive_connection#s`` :
`Pool limit configuration`_, overwrites value ``pool_maxsize`` from
:ref:`settings outgoing` for this engine.
``max_connections`` :
`Pool limit configuration`_, overwrites value ``pool_connections`` from
:ref:`settings outgoing` for this engine.
``keepalive_expiry`` :
`Pool limit configuration`_, overwrites value ``keepalive_expiry`` from
:ref:`settings outgoing` for this engine.
.. _private engines:
Private Engines (``tokens``)
============================
Administrators might find themselves wanting to limit access to some of the
enabled engines on their instances. It might be because they do not want to
expose some private information through :ref:`offline engines`. Or they would
rather share engines only with their trusted friends or colleagues.
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
To solve this issue the concept of *private engines* exists.
A new option was added to engines named `tokens`. It expects a list of strings.
If the user making a request presents one of the tokens of an engine, they can
access information about the engine and make search requests.
Example configuration to restrict access to the Arch Linux Wiki engine:
.. code:: yaml
- name: arch linux wiki
engine: archlinux
shortcut: al
tokens: [ 'my-secret-token' ]
Unless a user has configured the right token, the engine is going to be hidden
from them. It is not going to be included in the list of engines on the
Preferences page and in the output of `/config` REST API call.
Tokens can be added to one's configuration on the Preferences page under "Engine
tokens". The input expects a comma separated list of strings.
The distribution of the tokens from the administrator to the users is not carved
in stone. As providing access to such engines implies that the admin knows and
trusts the user, we do not see necessary to come up with a strict process.
Instead, we would like to add guidelines to the documentation of the feature.
Example: Multilingual Search
============================
SearXNG does not support true multilingual search. You have to use the language
prefix in your search query when searching in a different language.
But there is a workaround: By adding a new search engine with a different
language, SearXNG will search in your default and other language.
Example configuration in settings.yml for a German and English speaker:
.. code-block:: yaml
search:
default_lang : "de"
...
engines:
- name : google english
engine : google
language : en
...
When searching, the default google engine will return German results and
"google english" will return English results.

View File

@ -1,34 +0,0 @@
.. _settings general:
============
``general:``
============
.. code:: yaml
general:
debug: false
instance_name: "SearXNG"
privacypolicy_url: false
donation_url: false
contact_url: false
enable_metrics: true
``debug`` : ``$SEARXNG_DEBUG``
Allow a more detailed log if you run SearXNG directly. Display *detailed* error
messages in the browser too, so this must be deactivated in production.
``donation_url`` :
Set value to ``true`` to use your own donation page written in the
:ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable
the donation link altogether.
``privacypolicy_url``:
Link to privacy policy.
``contact_url``:
Contact ``mailto:`` address or WEB form.
``enable_metrics``:
Enabled by default. Record various anonymous metrics available at ``/stats``,
``/stats/errors`` and ``/preferences``.

View File

@ -1,110 +0,0 @@
.. _settings outgoing:
=============
``outgoing:``
=============
Communication with search engines.
.. code:: yaml
outgoing:
request_timeout: 2.0 # default timeout in seconds, can be override by engine
max_request_timeout: 10.0 # the maximum timeout in seconds
useragent_suffix: "" # information like an email address to the administrator
pool_connections: 100 # Maximum number of allowable connections, or null
# for no limits. The default is 100.
pool_maxsize: 10 # Number of allowable keep-alive connections, or null
# to always allow. The default is 10.
enable_http2: true # See https://www.python-httpx.org/http2/
# uncomment below section if you want to use a custom server certificate
# see https://www.python-httpx.org/advanced/#changing-the-verification-defaults
# and https://www.python-httpx.org/compatibility/#ssl-configuration
# verify: ~/.mitmproxy/mitmproxy-ca-cert.cer
#
# uncomment below section if you want to use a proxyq see: SOCKS proxies
# https://2.python-requests.org/en/latest/user/advanced/#proxies
# are also supported: see
# https://2.python-requests.org/en/latest/user/advanced/#socks
#
# proxies:
# all://:
# - http://proxy1:8080
# - http://proxy2:8080
#
# using_tor_proxy: true
#
# Extra seconds to add in order to account for the time taken by the proxy
#
# extra_proxy_timeout: 10.0
#
``request_timeout`` :
Global timeout of the requests made to others engines in seconds. A bigger
timeout will allow to wait for answers from slow engines, but in consequence
will slow SearXNG reactivity (the result page may take the time specified in the
timeout to load). Can be override by ``timeout`` in the :ref:`settings engine`.
``useragent_suffix`` :
Suffix to the user-agent SearXNG uses to send requests to others engines. If an
engine wish to block you, a contact info here may be useful to avoid that.
.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration
``pool_maxsize``:
Number of allowable keep-alive connections, or ``null`` to always allow. The
default is 10. See ``max_keepalive_connections`` `Pool limit configuration`_.
``pool_connections`` :
Maximum number of allowable connections, or ``null`` # for no limits. The
default is 100. See ``max_connections`` `Pool limit configuration`_.
``keepalive_expiry`` :
Number of seconds to keep a connection in the pool. By default 5.0 seconds.
See ``keepalive_expiry`` `Pool limit configuration`_.
.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying
``proxies`` :
Define one or more proxies you wish to use, see `httpx proxies`_.
If there are more than one proxy for one protocol (http, https),
requests to the engines are distributed in a round-robin fashion.
``source_ips`` :
If you use multiple network interfaces, define from which IP the requests must
be made. Example:
* ``0.0.0.0`` any local IPv4 address.
* ``::`` any local IPv6 address.
* ``192.168.0.1``
* ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses
* ``fe80::60a2:1691:e5a2:ee1f``
* ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.
* ``[ 192.168.0.1, fe80::/126 ]``
``retries`` :
Number of retry in case of an HTTP error. On each retry, SearXNG uses an
different proxy and source ip.
``enable_http2`` :
Enable by default. Set to ``false`` to disable HTTP/2.
.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults
.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration
``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR``
Allow to specify a path to certificate.
see `httpx verification defaults`_.
In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and
``$SSL_CERT_DIR`` (for a directory) OpenSSL variables.
see `httpx ssl configuration`_.
``max_redirects`` :
30 by default. Maximum redirect before it is an error.
``using_tor_proxy`` :
Using tor proxy (``true``) or not (``false``) for all engines. The default is
``false`` and can be overwritten in the :ref:`settings engine`

View File

@ -1,49 +0,0 @@
.. _settings redis:
==========
``redis:``
==========
.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url
A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you
will find a description to test your redis connection in SearXNG. When using
sockets, don't forget to check the access rights on the socket::
ls -la /usr/local/searxng-redis/run/redis.sock
srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock
In this example read/write access is given to the *searxng-redis* group. To get
access rights to redis instance (the socket), your SearXNG (or even your
developer) account needs to be added to the *searxng-redis* group.
``url`` : ``$SEARXNG_REDIS_URL``
URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`::
redis://[[username]:[password]]@localhost:6379/0
rediss://[[username]:[password]]@localhost:6379/0
unix://[[username]:[password]]@/path/to/socket.sock?db=0
.. _Redis Developer Notes:
Redis Developer Notes
=====================
To set up a local redis instance, first set the socket path of the Redis DB
in your YAML setting:
.. code:: yaml
redis:
url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
Then use the following commands to install the redis instance (:ref:`manage
redis.help`):
.. code:: sh
$ ./manage redis.build
$ sudo -H ./manage redis.install
$ sudo -H ./manage redis.addgrp "${USER}"
# don't forget to logout & login to get member of group

View File

@ -1,104 +0,0 @@
.. _settings search:
===========
``search:``
===========
.. code:: yaml
search:
safe_search: 0
autocomplete: ""
favicon_resolver: ""
default_lang: ""
ban_time_on_fail: 5
max_ban_time_on_fail: 120
suspended_times:
SearxEngineAccessDenied: 86400
SearxEngineCaptcha: 86400
SearxEngineTooManyRequests: 3600
cf_SearxEngineCaptcha: 1296000
cf_SearxEngineAccessDenied: 86400
recaptcha_SearxEngineCaptcha: 604800
formats:
- html
``safe_search``:
Filter results.
- ``0``: None
- ``1``: Moderate
- ``2``: Strict
``autocomplete``:
Existing autocomplete backends, leave blank to turn it off.
- ``dbpedia``
- ``duckduckgo``
- ``google``
- ``mwmbl``
- ``startpage``
- ``swisscows``
- ``qwant``
- ``wikipedia``
``favicon_resolver``:
To activate favicons in SearXNG's result list select a default
favicon-resolver, leave blank to turn off the feature. Don't activate the
favicons before reading the :ref:`Favicons documentation <favicons>`.
``default_lang``:
Default search language - leave blank to detect from browser information or
use codes from :origin:`searx/languages.py`.
``languages``:
List of available languages - leave unset to use all codes from
:origin:`searx/languages.py`. Otherwise list codes of available languages.
The ``all`` value is shown as the ``Default language`` in the user interface
(in most cases, it is meant to send the query without a language parameter ;
in some cases, it means the English language) Example:
.. code:: yaml
languages:
- all
- en
- en-US
- de
- it-IT
- fr
- fr-BE
``ban_time_on_fail``:
Ban time in seconds after engine errors.
``max_ban_time_on_fail``:
Max ban time in seconds after engine errors.
``suspended_times``:
Engine suspension time after error (in seconds; set to 0 to disable)
``SearxEngineAccessDenied``: 86400
For error "Access denied" and "HTTP error [402, 403]"
``SearxEngineCaptcha``: 86400
For error "CAPTCHA"
``SearxEngineTooManyRequests``: 3600
For error "Too many request" and "HTTP error 429"
Cloudflare CAPTCHA:
- ``cf_SearxEngineCaptcha``: 1296000
- ``cf_SearxEngineAccessDenied``: 86400
Google CAPTCHA:
- ``recaptcha_SearxEngineCaptcha``: 604800
``formats``:
Result formats available from web, remove format to deny access (use lower
case).
- ``html``
- ``csv``
- ``json``
- ``rss``

View File

@ -1,57 +0,0 @@
.. _settings server:
===========
``server:``
===========
.. code:: yaml
server:
base_url: http://example.org/location # change this!
port: 8888
bind_address: "127.0.0.1"
secret_key: "ultrasecretkey" # change this!
limiter: false
public_instance: false
image_proxy: false
default_http_headers:
X-Content-Type-Options : nosniff
X-Download-Options : noopen
X-Robots-Tag : noindex, nofollow
Referrer-Policy : no-referrer
``base_url`` : ``$SEARXNG_URL``
The base URL where SearXNG is deployed. Used to create correct inbound links.
``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS``
Port number and *bind address* of the SearXNG web application if you run it
directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG
services running behind a proxy and using socket communications.
``secret_key`` : ``$SEARXNG_SECRET``
Used for cryptography purpose.
``limiter`` : ``$SEARXNG_LIMITER``
Rate limit the number of request on the instance, block some bots. The
:ref:`limiter` requires a :ref:`settings redis` database.
.. _public_instance:
``public_instance`` : ``$SEARXNG_PUBLIC_INSTANCE``
Setting that allows to enable features specifically for public instances (not
needed for local usage). By set to ``true`` the following features are
activated:
- :py:obj:`searx.botdetection.link_token` in the :ref:`limiter`
.. _image_proxy:
``image_proxy`` : ``$SEARXNG_IMAGE_PROXY``
Allow your instance of SearXNG of being able to proxy images. Uses memory space.
.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
``default_http_headers`` :
Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__

View File

@ -1,70 +0,0 @@
.. _settings ui:
=======
``ui:``
=======
.. _cache busting:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting
.. code:: yaml
ui:
static_use_hash: false
default_locale: ""
query_in_title: false
infinite_scroll: false
center_alignment: false
cache_url: https://web.archive.org/web/
default_theme: simple
theme_args:
simple_style: auto
search_on_category_select: true
hotkeys: default
.. _static_use_hash:
``static_use_hash`` : ``$SEARXNG_STATIC_USE_HASH``
Enables `cache busting`_ of static files.
``default_locale`` :
SearXNG interface language. If blank, the locale is detected by using the
browser language. If it doesn't work, or you are deploying a language
specific instance of searx, a locale can be defined using an ISO language
code, like ``fr``, ``en``, ``de``.
``query_in_title`` :
When true, the result page's titles contains the query it decreases the
privacy, since the browser can records the page titles.
``infinite_scroll``:
When true, automatically loads the next page when scrolling to bottom of the current page.
``center_alignment`` : default ``false``
When enabled, the results are centered instead of being in the left (or RTL)
side of the screen. This setting only affects the *desktop layout*
(:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`)
.. cache_url:
``cache_url`` : ``https://web.archive.org/web/``
URL prefix of the internet archive or cache, don't forget trailing slash (if
needed). The default is https://web.archive.org/web/ alternatives are:
- https://webcache.googleusercontent.com/search?q=cache:
- https://archive.today/
``default_theme`` :
Name of the theme you want to use by default on your SearXNG instance.
``theme_args.simple_style``:
Style of simple theme: ``auto``, ``light``, ``dark``, ``black``
``results_on_new_tab``:
Open result links in a new tab by default.
``search_on_category_select``:
Perform search immediately if a category selected. Disable to select multiple categories.
``hotkeys``:
Hotkeys to use in the search interface: ``default``, ``vim`` (Vim-like).

View File

@ -9,7 +9,7 @@ SearXNG maintenance
- :ref:`toolboxing` - :ref:`toolboxing`
- :ref:`uWSGI maintenance` - :ref:`uWSGI maintenance`
.. contents:: .. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
@ -61,7 +61,7 @@ and then, to name just a few:
- The image proxy morty is no longer needed, it has been replaced by the - The image proxy morty is no longer needed, it has been replaced by the
:ref:`image proxy <image_proxy>` from SearXNG. :ref:`image proxy <image_proxy>` from SearXNG.
- To save bandwidth :ref:`cache busting <static_use_hash>` has been implemented. - To save bandwith :ref:`cache busting <static_use_hash>` has been implemented.
To get in use, the ``static-expires`` needs to be set in the :ref:`uwsgi To get in use, the ``static-expires`` needs to be set in the :ref:`uwsgi
setup`. setup`.

View File

@ -0,0 +1,53 @@
.. START create user
.. tabs::
.. group-tab:: bash
.. code-block:: sh
$ sudo -H useradd --shell /bin/bash --system \\
--home-dir \"$SERVICE_HOME\" \\
--comment \"Privacy-respecting metasearch engine\" $SERVICE_USER
$ sudo -H mkdir \"$SERVICE_HOME\"
$ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
.. END create user
.. START install go
.. tabs::
.. group-tab:: os: linux / arch: amd64
.. code-block:: bash
$ cat > \"$GO_ENV\" <<EOF
export GOPATH=${SERVICE_HOME}/go-apps
export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin
EOF
$ sudo -i -u \"${SERVICE_USER}\"
(${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile
(${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local
(${SERVICE_USER}) $ wget --progress=bar -O \"${GO_VERSION}.linux-amd64.tar.gz\" \\
\"${GO_DL_URL}/${GO_VERSION}.linux-amd64.tar.gz\"
(${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_VERSION}.linux-amd64.tar.gz\"
(${SERVICE_USER}) $ which go
${SERVICE_HOME}/local/go/bin/go
.. END install go
.. START install filtron
.. tabs::
.. group-tab:: bash
.. code-block:: bash
$ sudo -i -u \"${SERVICE_USER}\"
(${SERVICE_USER}) $ go get -v -u github.com/searxng/filtron
.. END install filtron

View File

@ -0,0 +1,53 @@
.. START create user
.. tabs::
.. group-tab:: bash
.. code-block:: sh
$ sudo -H useradd --shell /bin/bash --system \\
--home-dir \"$SERVICE_HOME\" \\
--comment \"Privacy-respecting metasearch engine\" $SERVICE_USER
$ sudo -H mkdir \"$SERVICE_HOME\"
$ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
.. END create user
.. START install go
.. tabs::
.. group-tab:: os: linux / arch: amd64
.. code-block:: bash
$ cat > \"$GO_ENV\" <<EOF
export GOPATH=${SERVICE_HOME}/go-apps
export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin
EOF
$ sudo -i -u \"${SERVICE_USER}\"
(${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile
(${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local
(${SERVICE_USER}) $ wget --progress=bar -O \"${GO_VERSION}.linux-amd64.tar.gz\" \\
\"${GO_DL_URL}/${GO_VERSION}.linux-amd64.tar.gz\"
(${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_VERSION}.linux-amd64.tar.gz\"
(${SERVICE_USER}) $ which go
${SERVICE_HOME}/local/go/bin/go
.. END install go
.. START install morty
.. tabs::
.. group-tab:: bash
.. code-block:: bash
$ sudo -i -u \"${SERVICE_USER}\"
(${SERVICE_USER}) $ go get -v -u github.com/asciimoo/morty
.. END install morty

View File

@ -113,7 +113,7 @@ ${fedora_build}
(${SERVICE_USER})$ command -v python && python --version (${SERVICE_USER})$ command -v python && python --version
$SEARXNG_PYENV/bin/python $SEARXNG_PYENV/bin/python
Python 3.11.10 Python 3.8.1
# update pip's boilerplate .. # update pip's boilerplate ..
pip install -U pip pip install -U pip
@ -123,7 +123,7 @@ ${fedora_build}
# jump to SearXNG's working tree and install SearXNG into virtualenv # jump to SearXNG's working tree and install SearXNG into virtualenv
(${SERVICE_USER})$ cd \"$SEARXNG_SRC\" (${SERVICE_USER})$ cd \"$SEARXNG_SRC\"
(${SERVICE_USER})$ pip install --use-pep517 --no-build-isolation -e . (${SERVICE_USER})$ pip install -e .
.. END manage.sh update_packages .. END manage.sh update_packages

View File

@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
# SPDX-License-Identifier: AGPL-3.0-or-later # SPDX-License-Identifier: AGPL-3.0-or-later
import sys, os import sys, os
from pathlib import Path
from pallets_sphinx_themes import ProjectLink from pallets_sphinx_themes import ProjectLink
from searx import get_setting from searx import get_setting
@ -10,9 +10,10 @@ from searx.version import VERSION_STRING, GIT_URL, GIT_BRANCH
# Project -------------------------------------------------------------- # Project --------------------------------------------------------------
project = 'SearXNG' project = 'SearXNG'
copyright = 'SearXNG team' copyright = '2021 SearXNG team, 2015-2021 Adam Tauber, Noémi Ványi'
author = 'SearXNG team' author = '2021 SearXNG team, 2015-2021 Adam Tauber'
release, version = VERSION_STRING, VERSION_STRING release, version = VERSION_STRING, VERSION_STRING
SEARXNG_URL = get_setting('server.base_url') or 'https://example.org/searxng' SEARXNG_URL = get_setting('server.base_url') or 'https://example.org/searxng'
ISSUE_URL = get_setting('brand.issue_url') ISSUE_URL = get_setting('brand.issue_url')
DOCS_URL = get_setting('brand.docs_url') DOCS_URL = get_setting('brand.docs_url')
@ -21,9 +22,6 @@ PRIVACYPOLICY_URL = get_setting('general.privacypolicy_url')
CONTACT_URL = get_setting('general.contact_url') CONTACT_URL = get_setting('general.contact_url')
WIKI_URL = get_setting('brand.wiki_url') WIKI_URL = get_setting('brand.wiki_url')
SOURCEDIR = Path(__file__).parent.parent / "searx"
os.environ['SOURCEDIR'] = str(SOURCEDIR)
# hint: sphinx.ext.viewcode won't highlight when 'highlight_language' [1] is set # hint: sphinx.ext.viewcode won't highlight when 'highlight_language' [1] is set
# to string 'none' [2] # to string 'none' [2]
# #
@ -127,7 +125,6 @@ extensions = [
"sphinx_tabs.tabs", # https://github.com/djungelorm/sphinx-tabs "sphinx_tabs.tabs", # https://github.com/djungelorm/sphinx-tabs
'myst_parser', # https://www.sphinx-doc.org/en/master/usage/markdown.html 'myst_parser', # https://www.sphinx-doc.org/en/master/usage/markdown.html
'notfound.extension', # https://github.com/readthedocs/sphinx-notfound-page 'notfound.extension', # https://github.com/readthedocs/sphinx-notfound-page
'sphinxcontrib.autodoc_pydantic', # https://github.com/mansenfranzen/autodoc_pydantic
] ]
autodoc_default_options = { autodoc_default_options = {
@ -171,7 +168,6 @@ imgmath_image_format = 'svg'
imgmath_font_size = 14 imgmath_font_size = 14
# sphinx.ext.imgmath setup END # sphinx.ext.imgmath setup END
html_show_sphinx = False
html_theme_options = {"index_sidebar_logo": True} html_theme_options = {"index_sidebar_logo": True}
html_context = {"project_links": [] } html_context = {"project_links": [] }
html_context["project_links"].append(ProjectLink("Source", GIT_URL + '/tree/' + GIT_BRANCH)) html_context["project_links"].append(ProjectLink("Source", GIT_URL + '/tree/' + GIT_BRANCH))

View File

@ -4,7 +4,7 @@
How to contribute How to contribute
================= =================
.. contents:: .. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
@ -148,7 +148,7 @@ live build
Live build is like WYSIWYG. If you want to edit the documentation, its Live build is like WYSIWYG. If you want to edit the documentation, its
recommended to use. The Makefile target ``docs.live`` builds the docs, opens recommended to use. The Makefile target ``docs.live`` builds the docs, opens
URL in your favorite browser and rebuilds every time a reST file has been URL in your favorite browser and rebuilds every time a reST file has been
changed (:ref:`make docs.clean`). changed.
.. code:: sh .. code:: sh
@ -183,8 +183,3 @@ commit and push:
.. code:: sh .. code:: sh
$ make docs.clean docs.gh-pages $ make docs.clean docs.gh-pages
.. attention::
If you are working in your own brand, don't forget to adjust your
:ref:`settings brand`.

View File

@ -4,11 +4,6 @@
Engine Overview Engine Overview
=============== ===============
.. contents::
:depth: 3
:local:
:backlinks: entry
.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine .. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
.. sidebar:: Further reading .. .. sidebar:: Further reading ..
@ -16,6 +11,10 @@ Engine Overview
- :ref:`configured engines` - :ref:`configured engines`
- :ref:`settings engine` - :ref:`settings engine`
.. contents::
:depth: 3
:backlinks: entry
SearXNG is a metasearch-engine_, so it uses different search engines to provide SearXNG is a metasearch-engine_, so it uses different search engines to provide
better results. better results.
@ -50,12 +49,12 @@ Engine File
categories list categories, in which the engine is working categories list categories, in which the engine is working
paging boolean support multiple pages paging boolean support multiple pages
time_range_support boolean support search time range time_range_support boolean support search time range
engine_type str - ``online`` :ref:`[ref] <online engines>` by engine_type str - ``online`` :ref:`[ref] <demo online engine>` by
default, other possibles values are: default, other possibles values are:
- ``offline`` :ref:`[ref] <offline engines>` - ``offline`` :ref:`[ref] <offline engines>`
- ``online_dictionary`` :ref:`[ref] <online dictionary>` - ``online_dictionary``
- ``online_currency`` :ref:`[ref] <online currency>` - ``online_currency``
- ``online_url_search`` :ref:`[ref] <online url search>` - ``online_url_search``
======================= =========== ======================================================== ======================= =========== ========================================================
.. _engine settings: .. _engine settings:
@ -87,8 +86,8 @@ For a more detailed description, see :ref:`settings engine` in the :ref:`settin
Overrides Overrides
--------- ---------
A few of the options have default values in the namespace of the engine's python A few of the options have default values in the namespace of engine's python
module, but are often overwritten by the settings. If ``None`` is assigned to an modul, but are often overwritten by the settings. If ``None`` is assigned to an
option in the engine file, it has to be redefined in the settings, otherwise option in the engine file, it has to be redefined in the settings, otherwise
SearXNG will not start with that engine (global names with a leading underline can SearXNG will not start with that engine (global names with a leading underline can
be ``None``). be ``None``).
@ -240,18 +239,12 @@ following parameters can be used to specify a search request:
.. _engine results: .. _engine results:
.. _engine media types: .. _engine media types:
Result Types (``template``) Media Types
=========================== ===========
Each result item of an engine can be of different media-types. Currently the Each result item of an engine can be of different media-types. Currently the
following media-types are supported. To set another media-type as following media-types are supported. To set another media-type as ``default``,
:ref:`template default`, the parameter ``template`` must be set to the desired the parameter ``template`` must be set to the desired type.
type.
.. _template default:
``default``
-----------
.. table:: Parameter of the **default** media type: .. table:: Parameter of the **default** media type:
:width: 100% :width: 100%
@ -266,65 +259,23 @@ type.
========================= ===================================================== ========================= =====================================================
.. _template images: .. table:: Parameter of the **images** media type:
``images``
----------
.. list-table:: Parameter of the **images** media type
:header-rows: 2
:width: 100% :width: 100%
* - result-parameter ========================= =====================================================
- Python type result-parameter information
- information ------------------------- -----------------------------------------------------
template is set to ``images.html``
========================= =====================================================
url string, url to the result site
title string, title of the result *(partly implemented)*
content *(partly implemented)*
publishedDate :py:class:`datetime.datetime`,
time of publish *(partly implemented)*
img\_src string, url to the result image
thumbnail\_src string, url to a small-preview image
========================= =====================================================
* - template
- :py:class:`str`
- is set to ``images.html``
* - url
- :py:class:`str`
- url to the result site
* - title
- :py:class:`str`
- title of the result
* - content
- :py:class:`str`
- description of the image
* - publishedDate
- :py:class:`datetime <datetime.datetime>`
- time of publish
* - img_src
- :py:class:`str`
- url to the result image
* - thumbnail_src
- :py:class:`str`
- url to a small-preview image
* - resolution
- :py:class:`str`
- the resolution of the image (e.g. ``1920 x 1080`` pixel)
* - img_format
- :py:class:`str`
- the format of the image (e.g. ``png``)
* - filesize
- :py:class:`str`
- size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation
(e.g. ``MB`` for 1024 \* 1024 Bytes filesize).
.. _template videos:
``videos``
----------
.. table:: Parameter of the **videos** media type: .. table:: Parameter of the **videos** media type:
:width: 100% :width: 100%
@ -339,16 +290,8 @@ type.
content *(not implemented yet)* content *(not implemented yet)*
publishedDate :py:class:`datetime.datetime`, time of publish publishedDate :py:class:`datetime.datetime`, time of publish
thumbnail string, url to a small-preview image thumbnail string, url to a small-preview image
length :py:class:`datetime.timedelta`, duration of result
views string, view count in humanized number format
========================= ===================================================== ========================= =====================================================
.. _template torrent:
``torrent``
-----------
.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme .. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
.. table:: Parameter of the **torrent** media type: .. table:: Parameter of the **torrent** media type:
@ -372,12 +315,6 @@ type.
torrentfile string, torrentfile of the result torrentfile string, torrentfile of the result
========================= ===================================================== ========================= =====================================================
.. _template map:
``map``
-------
.. table:: Parameter of the **map** media type: .. table:: Parameter of the **map** media type:
:width: 100% :width: 100%
@ -405,12 +342,6 @@ type.
address.country country of object address.country country of object
========================= ===================================================== ========================= =====================================================
.. _template paper:
``paper``
---------
.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ .. _BibTeX format: https://www.bibtex.com/g/bibtex-format/
.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types .. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types
@ -499,73 +430,3 @@ type.
* - html_url * - html_url
- :py:class:`str` - :py:class:`str`
- URL to full article, HTML version - URL to full article, HTML version
.. _template packages:
``packages``
------------
.. list-table:: Parameter of the **packages** media type
:header-rows: 2
:width: 100%
* - result-parameter
- Python type
- information
* - template
- :py:class:`str`
- is set to ``packages.html``
* - title
- :py:class:`str`
- title of the result
* - content
- :py:class:`str`
- abstract
* - package_name
- :py:class:`str`
- the name of the package
* - version
- :py:class:`str`
- the current version of the package
* - maintainer
- :py:class:`str`
- the maintainer or author of the project
* - publishedDate
- :py:class:`datetime <datetime.datetime>`
- date of latest update or release
* - tags
- :py:class:`List <list>`\ [\ :py:class:`str`\ ]
- free tag list
* - popularity
- :py:class:`str`
- the popularity of the package, e.g. rating or download count
* - license_name
- :py:class:`str`
- the name of the license
* - license_url
- :py:class:`str`
- the web location of a license copy
* - homepage
- :py:class:`str`
- the url of the project's homepage
* - source_code_url
- :py:class:`str`
- the location of the project's source code
* - links
- :py:class:`dict`
- additional links in the form of ``{'link_name': 'http://example.com'}``

View File

@ -1,9 +0,0 @@
.. _searx.engines loader:
========================
SearXNG's engines loader
========================
.. automodule:: searx.engines
:members:

View File

@ -1,107 +0,0 @@
.. _engine implementations:
======================
Engine Implementations
======================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. toctree::
:caption: Framework Components
:maxdepth: 2
enginelib
engines
engine_overview
Engine Types
============
The :py:obj:`engine_type <searx.enginelib.Engine.engine_type>` of an engine
determines which :ref:`search processor <searx.search.processors>` is used by
the engine.
In this section a list of the engines that are documented is given, a complete
list of the engines can be found in the source under: :origin:`searx/engines`.
.. _online engines:
Online Engines
--------------
.. sidebar:: info
- :py:obj:`processors.online <searx.search.processors.online>`
.. toctree::
:maxdepth: 1
:glob:
demo/demo_online
xpath
mediawiki
.. toctree::
:maxdepth: 1
:glob:
online/*
.. _offline engines:
Offline Engines
---------------
.. sidebar:: info
- :py:obj:`processors.offline <searx.search.processors.offline>`
.. toctree::
:maxdepth: 1
:glob:
offline_concept
demo/demo_offline
offline/*
.. _online url search:
Online URL Search
-----------------
.. sidebar:: info
- :py:obj:`processors.online_url_search <searx.search.processors.online_url_search>`
.. toctree::
:maxdepth: 1
:glob:
online_url_search/*
.. _online currency:
Online Currency
---------------
.. sidebar:: info
- :py:obj:`processors.online_currency <searx.search.processors.online_currency>`
*no engine of this type is documented yet / comming soon*
.. _online dictionary:
Online Dictionary
-----------------
.. sidebar:: info
- :py:obj:`processors.online_dictionary <searx.search.processors.online_dictionary>`
*no engine of this type is documented yet / comming soon*

View File

@ -1,13 +0,0 @@
.. _mediawiki engine:
================
MediaWiki Engine
================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.mediawiki
:members:

View File

@ -1,23 +0,0 @@
.. _engine command:
====================
Command Line Engines
====================
.. sidebar:: info
- :origin:`command.py <searx/engines/command.py>`
- :ref:`offline engines`
.. contents::
:depth: 2
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
.. automodule:: searx.engines.command
:members:

View File

@ -1,62 +0,0 @@
=================
Local Search APIs
=================
.. sidebar:: further read
- `Comparison to alternatives
<https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_
.. contents::
:depth: 1
:local:
:backlinks: entry
.. sidebar:: info
Initial sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_.
Administrators might find themselves wanting to integrate locally running search
engines. The following ones are supported for now:
* `Elasticsearch`_
* `Meilisearch`_
* `Solr`_
Each search engine is powerful, capable of full-text search. All of the engines
above are added to ``settings.yml`` just commented out, as you have to
``base_url`` for all them.
Please note that if you are not using HTTPS to access these engines, you have to
enable HTTP requests by setting ``enable_http`` to ``True``.
Furthermore, if you do not want to expose these engines on a public instance,
you can still add them and limit the access by setting ``tokens`` as described
in section :ref:`private engines`.
.. _engine meilisearch:
MeiliSearch
===========
.. automodule:: searx.engines.meilisearch
:members:
.. _engine elasticsearch:
Elasticsearch
=============
.. automodule:: searx.engines.elasticsearch
:members:
.. _engine solr:
Solr
====
.. automodule:: searx.engines.solr
:members:

View File

@ -1,13 +0,0 @@
.. _alpinelinux engine:
=====================
Alpine Linux Packages
=====================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.alpinelinux
:members:

View File

@ -1,13 +0,0 @@
.. _annas_archive engine:
==============
Anna's Archive
==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.annas_archive
:members:

View File

@ -1,13 +0,0 @@
.. _bpb engine:
===
Bpb
===
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.bpb
:members:

View File

@ -1,13 +0,0 @@
.. _brave engine:
=============
Brave Engines
=============
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.brave
:members:

View File

@ -1,14 +0,0 @@
.. _bt4g engine:
====
BT4G
====
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.bt4g
:members:

View File

@ -1,8 +0,0 @@
.. _discourse engine:
================
Discourse Forums
================
.. automodule:: searx.engines.discourse
:members:

View File

@ -1,8 +0,0 @@
.. _gitea geizhals:
========
Geizhals
========
.. automodule:: searx.engines.geizhals
:members:

View File

@ -1,8 +0,0 @@
.. _gitea engine:
=====
Gitea
=====
.. automodule:: searx.engines.gitea
:members:

View File

@ -1,8 +0,0 @@
.. _gitlab engine:
======
GitLab
======
.. automodule:: searx.engines.gitlab
:members:

View File

@ -1,13 +0,0 @@
.. _lemmy engine:
=====
Lemmy
=====
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.lemmy
:members:

View File

@ -1,13 +0,0 @@
.. _loc engine:
===================
Library of Congress
===================
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.loc
:members:

View File

@ -1,13 +0,0 @@
.. _mastodon engine:
========
Mastodon
========
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.mastodon
:members:

View File

@ -1,13 +0,0 @@
.. _moviepilot engine:
==========
Moviepilot
==========
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.moviepilot
:members:

View File

@ -1,13 +0,0 @@
.. _mrs engine:
=========================
Matrix Rooms Search (MRS)
=========================
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.mrs
:members:

View File

@ -1,13 +0,0 @@
.. _voidlinux mullvad_leta:
============
Mullvad-Leta
============
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.mullvad_leta
:members:

View File

@ -1,27 +0,0 @@
.. _Mwmbl engine:
============
Mwmbl Engine
============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. _mwmbl web engine:
Mwmbl WEB
=========
.. automodule:: searx.engines.mwmbl
:members:
.. _mwmbl autocomplete:
Mwmbl Autocomplete
==================
.. autofunction:: searx.autocomplete.mwmbl

View File

@ -1,13 +0,0 @@
.. _odysee engine:
======
Odysee
======
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.odysee
:members:

View File

@ -1,13 +0,0 @@
.. _piped engine:
=====
Piped
=====
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.piped
:members:

View File

@ -1,13 +0,0 @@
.. _engine presearch:
================
Presearch Engine
================
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.presearch
:members:

View File

@ -1,13 +0,0 @@
.. _qwant engine:
=====
Qwant
=====
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.qwant
:members:

View File

@ -1,13 +0,0 @@
.. _RadioBrowser engine:
============
RadioBrowser
============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.radio_browser
:members:

View File

@ -1,13 +0,0 @@
.. _engine recoll:
=============
Recoll Engine
=============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.recoll
:members:

View File

@ -1,13 +0,0 @@
.. _seekr engine:
=============
Seekr Engines
=============
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.seekr
:members:

View File

@ -1,13 +0,0 @@
.. _tagesschau engine:
==============
Tagesschau API
==============
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.tagesschau
:members:

View File

@ -1,13 +0,0 @@
.. _torznab engine:
==============
Torznab WebAPI
==============
.. contents::
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.torznab
:members:

View File

@ -1,13 +0,0 @@
.. _voidlinux engine:
==========================
Void Linux binary packages
==========================
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.voidlinux
:members:

View File

@ -1,13 +0,0 @@
.. _wallhaven engine:
=========
Wallhaven
=========
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.wallhaven
:members:

View File

@ -1,13 +0,0 @@
.. _yacy engine:
====
Yacy
====
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.yacy
:members:

View File

@ -1,13 +0,0 @@
.. _zlibrary engine:
=========
Z-Library
=========
.. contents:: Contents
:depth: 2
:local:
:backlinks: entry
.. automodule:: searx.engines.zlibrary
:members:

View File

@ -4,11 +4,12 @@ Developer documentation
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Contents
quickstart quickstart
rtm_asdf
contribution_guide contribution_guide
engines/index engine_overview
offline_engines
search_api search_api
plugins plugins
translation translation

View File

@ -22,7 +22,7 @@ In this article we will show, how you can make use of Linux Containers (LXC_) in
section :ref:`internet connectivity docker`. section :ref:`internet connectivity docker`.
.. contents:: .. contents:: Contents
:depth: 2 :depth: 2
:local: :local:
:backlinks: entry :backlinks: entry
@ -148,7 +148,7 @@ services quite easy.
In the example above the SearXNG instance in the container is wrapped to In the example above the SearXNG instance in the container is wrapped to
``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the ``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the
IP of your ``searxng-archlinux`` container is different to this example. To IP of your ``searxng-archlinux`` container is different to this example. To
test the instance in the container from outside of the container, in your WEB test the instance in the conatiner from outside of the container, in your WEB
browser on your desktop just open the URL reported in your installation browser on your desktop just open the URL reported in your installation
.. _working in containers: .. _working in containers:
@ -255,7 +255,7 @@ With the use of the :ref:`searxng.sh` the SearXNG service was installed as
With the command above, we stopped the SearXNG uWSGI-App in the archlinux With the command above, we stopped the SearXNG uWSGI-App in the archlinux
container. container.
The uWSGI-App for the archlinux distros is configured in The uWSGI-App for the archlinux dsitros is configured in
:origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at :origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at
least you should attend the settings of ``uid``, ``chdir``, ``env`` and least you should attend the settings of ``uid``, ``chdir``, ``env`` and
``http``:: ``http``::
@ -272,7 +272,7 @@ shares the root folder of the repository and the command ``utils/lxc.sh cmd``
handles relative path names **transparent**. handles relative path names **transparent**.
To wrap the SearXNG installation in the container into a developer one, we To wrap the SearXNG installation in the container into a developer one, we
simple have to create a symlink to the **transparent** repository from the simple have to create a smylink to the **transparent** reposetory from the
desktop. Now lets replace the repository at ``searxng-src`` in the container desktop. Now lets replace the repository at ``searxng-src`` in the container
with the working tree from outside of the container: with the working tree from outside of the container:

View File

@ -1,48 +1,31 @@
.. _makefile: .. _makefile:
======================= ========
Makefile & ``./manage`` Makefile
======================= ========
.. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction .. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction
All relevant build and development tasks are implemented in the
:origin:`./manage <manage>` script and for CI or IDE integration a small
:origin:`Makefile` wrapper is available. If you are not familiar with
Makefiles, we recommend to read gnu-make_ introduction.
.. sidebar:: build environment .. sidebar:: build environment
Before looking deeper at the targets, first read about :ref:`make Before looking deeper at the targets, first read about :ref:`make
install`. install`.
To install developer requirements follow :ref:`buildhosts`. To install system requirements follow :ref:`buildhosts`.
All relevant build tasks are implemented in :origin:`manage` and for CI or
.. contents:: IDE integration a small ``Makefile`` wrapper is available. If you are not
:depth: 2 familiar with Makefiles, we recommend to read gnu-make_ introduction.
:local:
:backlinks: entry
The usage is simple, just type ``make {target-name}`` to *build* a target. The usage is simple, just type ``make {target-name}`` to *build* a target.
Calling the ``help`` target gives a first overview (``make help``): Calling the ``help`` target gives a first overview (``make help``):
.. tabs:: .. program-output:: bash -c "cd ..; make --no-print-directory help"
.. group-tab:: ``make`` .. contents:: Contents
:depth: 2
.. program-output:: bash -c "cd ..; make --no-print-directory help" :local:
:backlinks: entry
.. group-tab:: ``./manage``
The Makefile targets are implemented for comfort, if you can do without
tab-completion and need to have a more granular control, use
:origin:`manage` without the Makefile wrappers.
.. code:: sh
$ ./manage help
.. _make install: .. _make install:
@ -61,9 +44,13 @@ working tree and release a ``make install`` to get a virtualenv with a
$ make install $ make install
PYENV [virtualenv] installing ./requirements*.txt into local/py3 PYENV [virtualenv] installing ./requirements*.txt into local/py3
... ...
PYENV [install] pip install --use-pep517 --no-build-isolation -e 'searx[test]' PYENV OK
PYENV [install] pip install -e 'searx[test]'
... ...
Successfully installed searxng-2023.7.19+a446dea1b Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
If you release ``make install`` multiple times the installation will only If you release ``make install`` multiple times the installation will only
rebuild if the sha256 sum of the *requirement files* fails. With other words: rebuild if the sha256 sum of the *requirement files* fails. With other words:
@ -78,9 +65,13 @@ the check fails if you edit the requirements listed in
... ...
PYENV [virtualenv] installing ./requirements*.txt into local/py3 PYENV [virtualenv] installing ./requirements*.txt into local/py3
... ...
PYENV [install] pip install --use-pep517 --no-build-isolation -e 'searx[test]' PYENV OK
PYENV [install] pip install -e 'searx[test]'
... ...
Successfully installed searxng-2023.7.19+a446dea1b Successfully installed argparse-1.4.0 searx
BUILDENV INFO:searx:load the default settings from ./searx/settings.yml
BUILDENV INFO:searx:Initialisation done
BUILDENV build utils/brand.env
.. sidebar:: drop environment .. sidebar:: drop environment
@ -90,6 +81,67 @@ the check fails if you edit the requirements listed in
If you think, something goes wrong with your ./local environment or you change If you think, something goes wrong with your ./local environment or you change
the :origin:`setup.py` file, you have to call :ref:`make clean`. the :origin:`setup.py` file, you have to call :ref:`make clean`.
.. _make buildenv:
``make buildenv``
=================
Rebuild instance's environment with the modified settings from the
:ref:`settings brand` and :ref:`settings server` section of your
:ref:`settings.yml <settings location>`.
What is the :origin:`utils/brand.env` needed for and why do you need to rebuild
it if necessary?
Short answer: :ref:`installation and maintenance <searxng maintenance>`
scripts are running outside of instance's runtime environment and need some
values defined in the runtime environment.
All the SearXNG setups are centralized in the :ref:`settings.yml` file. This
setup is available as long we are in a *installed instance*. E.g. the
*installed instance* on the server or the *installed developer instance* at
``./local`` (the later one is created by a :ref:`make install <make install>` or
:ref:`make run <make run>`).
Tasks running outside of an *installed instance*, especially :ref:`installation
and maintenance <searxng maintenance>` tasks running at (pre-) installation time
do not have access to the SearXNG setup (from a *installed instance*). Those
tasks need a *build environment*.
The ``make buildenv`` target will update the *build environment* in:
- :origin:`utils/brand.env`
Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:
- ``SEARXNG_URL`` from :ref:`server.base_url <settings server>` (aka
``PUBLIC_URL``)
- ``SEARXNG_BIND_ADDRESS`` from :ref:`server.bind_address <settings server>`
- ``SEARXNG_PORT`` from :ref:`server.port <settings server>`
The ``GIT_URL`` and ``GIT_BRANCH`` in the origin:`utils/brand.env` file, are
readed from the git VCS and the branch that is checked out when ``make
buildenv`` command runs.
.. _brand:
**I would like to create my own brand, how should I proceed?**
Create a remote branch (``example.org``), checkout the remote branch (on your
local developer desktop) and in the :origin:`searx/settings.yml` file in the
:ref:`settings server` section set ``base_url``. Run ``make buildenv`` and
create a commit for your brand.
On your server you clone the branch (``example.org``) into your HOME folder
``~`` from where you run the :ref:`installation <installation>` and
:ref:`maintenance <searxng maintenance>` task.
To upgrade you brand, rebase on SearXNG's master branch (on your local
developer desktop), force push it to your remote branch. Go to your server, do
a force pull and run :ref:`sudo -H ./utils/searxng.sh instance update <update
searxng>`.
.. _make node.env: .. _make node.env:
Node.js environment (``make node.env``) Node.js environment (``make node.env``)
@ -106,36 +158,13 @@ Node.js environment (``make node.env``)
Manager) to install latest LTS of Node.js_ locally: there is no need to Manager) to install latest LTS of Node.js_ locally: there is no need to
install nvm_ or npm_ on your system. install nvm_ or npm_ on your system.
To install NVM_ and Node.js_ in once you can use :ref:`make nvm.nodejs`. Use ``make nvm.status`` to get the current status of you Node.js_ and nvm_ setup.
.. _make nvm: Here is the output you will typically get on a Ubuntu 20.04 system which serves
only a `no longer active <https://nodejs.org/en/about/releases/>`_ Release
`Node.js v10.19.0 <https://packages.ubuntu.com/focal/nodejs>`_.
NVM ``make nvm.install nvm.status`` ::
-----------------------------------
Use ``make nvm.status`` to get the current status of your Node.js_ and nvm_
setup.
.. tabs::
.. group-tab:: nvm.install
.. code:: sh
$ LANG=C make nvm.install
INFO: install (update) NVM at ./searxng/.nvm
INFO: clone: https://github.com/nvm-sh/nvm.git
|| Cloning into './searxng/.nvm'...
INFO: checkout v0.39.4
|| HEAD is now at 8fbf8ab v0.39.4
.. group-tab:: nvm.status (ubu2004)
Here is the output you will typically get on a Ubuntu 20.04 system which
serves only a `no longer active <https://nodejs.org/en/about/releases/>`_
Release `Node.js v10.19.0 <https://packages.ubuntu.com/focal/nodejs>`_.
.. code:: sh
$ make nvm.status $ make nvm.status
INFO: Node.js is installed at /usr/bin/node INFO: Node.js is installed at /usr/bin/node
@ -144,11 +173,14 @@ setup.
INFO: npm is installed at /usr/bin/npm INFO: npm is installed at /usr/bin/npm
INFO: npm is version 6.14.4 INFO: npm is version 6.14.4
WARN: NVM is not installed WARN: NVM is not installed
INFO: to install NVM and Node.js (LTS) use: manage nvm install --lts
To install you can also use :ref:`make nvm.nodejs`
.. _make nvm.nodejs: .. _make nvm.nodejs:
``make nvm.nodejs`` ``make nvm.nodejs``
------------------- ===================
Install latest Node.js_ LTS locally (uses nvm_):: Install latest Node.js_ LTS locally (uses nvm_)::
@ -181,29 +213,10 @@ sources of the theme need to be rebuild. You can do that by running::
$ make themes.all $ make themes.all
Alternatively to ``themes.all`` you can run *live builds* of the theme you are Alternatively to ``themes.all`` you can run *live builds* of the theme you are
modify (:ref:`make themes`):: modify::
$ LIVE_THEME=simple make run $ LIVE_THEME=simple make run
.. _make format.python:
``make format.python``
======================
Format Python source code using `Black code style`_. See ``$BLACK_OPTIONS``
and ``$BLACK_TARGETS`` in :origin:`Makefile`.
.. attention::
We stuck at Black 22.12.0, please read comment in PR `Bump black from 22.12.0
to 23.1.0`_
.. _Bump black from 22.12.0 to 23.1.0:
https://github.com/searxng/searxng/pull/2159#pullrequestreview-1284094735
.. _Black code style:
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html
.. _make clean: .. _make clean:
``make clean`` ``make clean``
@ -224,36 +237,20 @@ calling ``make clean`` stop all processes using the :ref:`make install` or
.. _make docs: .. _make docs:
``make docs`` ``make docs docs.autobuild docs.clean``
============= =======================================
Target ``docs`` builds the documentation:
.. code:: bash
$ make docs
HTML ./docs --> file://
DOCS build build/docs/includes
...
The HTML pages are in dist/docs.
.. _make docs.clean:
``make docs.clean docs.live``
----------------------------------
We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute / We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute /
Documentation <contrib docs>` section. If you want to edit the documentation Documentation <contrib docs>` section. If you want to edit the documentation
read our :ref:`make docs.live` section. If you are working in your own brand, read our :ref:`make docs.live` section. If you are working in your own brand,
adjust your :ref:`settings brand`. adjust your :ref:`settings global`.
.. _make docs.gh-pages: .. _make docs.gh-pages:
``make docs.gh-pages`` ``make docs.gh-pages``
---------------------- ======================
To deploy on github.io first adjust your :ref:`settings brand`. For any To deploy on github.io first adjust your :ref:`settings global`. For any
further read :ref:`deploy on github.io`. further read :ref:`deploy on github.io`.
.. _make test: .. _make test:
@ -264,17 +261,17 @@ further read :ref:`deploy on github.io`.
Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit`` Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit``
and ``test.robot``. You can run tests selective, e.g.:: and ``test.robot``. You can run tests selective, e.g.::
$ make test.pep8 test.unit test.shell $ make test.pep8 test.unit test.sh
TEST test.pep8 OK TEST test.pep8 OK
... ...
TEST test.unit OK TEST test.unit OK
... ...
TEST test.shell OK TEST test.sh OK
.. _make test.shell: .. _make test.shell:
``make test.shell`` ``make test.shell``
------------------- ===================
:ref:`sh lint` / if you have changed some bash scripting run this test before :ref:`sh lint` / if you have changed some bash scripting run this test before
commit. commit.
@ -282,7 +279,7 @@ commit.
.. _make test.pylint: .. _make test.pylint:
``make test.pylint`` ``make test.pylint``
-------------------- ====================
.. _Pylint: https://www.pylint.org/ .. _Pylint: https://www.pylint.org/
@ -292,8 +289,8 @@ found in project's root folder :origin:`.pylintrc`.
.. _make search.checker: .. _make search.checker:
``make search.checker.{engine name}`` ``search.checker.{engine name}``
===================================== ================================
To check all engines:: To check all engines::
@ -321,63 +318,3 @@ To filter out HTTP redirects (3xx_)::
https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
-- --
.. _make themes:
``make themes.*``
=================
.. sidebar:: further read
- :ref:`devquickstart`
The :origin:`Makefile` targets ``make theme.*`` cover common tasks to build the
theme(s). The ``./manage themes.*`` command line can be used to convenient run
common theme build tasks.
.. program-output:: bash -c "cd ..; ./manage themes.help"
To get live builds while modifying CSS & JS use (:ref:`make run`):
.. code:: sh
$ LIVE_THEME=simple make run
.. _make static.build:
``make static.build.*``
=======================
.. sidebar:: further read
- :ref:`devquickstart`
The :origin:`Makefile` targets ``static.build.*`` cover common tasks to build (a
commit of) the static files. The ``./manage static.build..*`` command line
can be used to convenient run common build tasks of the static files.
.. program-output:: bash -c "cd ..; ./manage static.help"
.. _manage redis.help:
``./manage redis.help``
=======================
The ``./manage redis.*`` command line can be used to convenient run common Redis
tasks (:ref:`Redis developer notes`).
.. program-output:: bash -c "cd ..; ./manage redis.help"
.. _manage go.help:
``./manage go.help``
====================
The ``./manage go.*`` command line can be used to convenient run common `go
(wiki)`_ tasks.
.. _go (wiki): https://en.wikipedia.org/wiki/Go_(programming_language)
.. program-output:: bash -c "cd ..; ./manage go.help"

View File

@ -1,14 +1,15 @@
.. _offline engines:
=============== ===============
Offline Concept Offline Engines
=============== ===============
.. sidebar:: offline engines .. sidebar:: offline engines
- :ref:`demo offline engine` - :ref:`demo offline engine`
- :ref:`engine command`
- :ref:`sql engines` - :ref:`sql engines`
- :ref:`nosql engines` - :ref:`engine command`
- :py:obj:`searx.search.processors.offline` - :origin:`Redis <searx/engines/redis_server.py>`
To extend the functionality of SearXNG, offline engines are going to be To extend the functionality of SearXNG, offline engines are going to be
introduced. An offline engine is an engine which does not need Internet introduced. An offline engine is an engine which does not need Internet
@ -30,6 +31,7 @@ Programming Interface
in advance. in advance.
:py:func:`search(query, params) <searx.engines.demo_offline.searc>` :py:func:`search(query, params) <searx.engines.demo_offline.searc>`
Each offline engine has a function named ``search``. This function is Each offline engine has a function named ``search``. This function is
responsible to perform a search and return the results in a presentable responsible to perform a search and return the results in a presentable
format. (Where *presentable* means presentable by the selected result format. (Where *presentable* means presentable by the selected result
@ -67,3 +69,10 @@ administrators can set token(s) for each of the :ref:`private engines`. If a
query contains a valid token, then SearXNG performs the requested private query contains a valid token, then SearXNG performs the requested private
search. If not, requests from an offline engines return errors. search. If not, requests from an offline engines return errors.
Acknowledgement
===============
This development was sponsored by `Search and Discovery Fund
<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_ .

View File

@ -19,6 +19,10 @@ Example plugin
description = 'This plugin extends the suggestions with the word "example"' description = 'This plugin extends the suggestions with the word "example"'
default_on = False # disabled by default default_on = False # disabled by default
js_dependencies = tuple() # optional, list of static js files
css_dependencies = tuple() # optional, list of static css files
# attach callback to the post search hook # attach callback to the post search hook
# request: flask request object # request: flask request object
# ctx: the whole local context of the post search hook # ctx: the whole local context of the post search hook
@ -36,7 +40,7 @@ SearXNG plugins from *The Green Web Foundation* `[ref]
.. code:: bash .. code:: bash
$ sudo utils/searxng.sh instance cmd bash -c $ sudo utils/searxng.sh instance cmd bash
(searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins
In the :ref:`settings.yml` activate the ``plugins:`` section and add module In the :ref:`settings.yml` activate the ``plugins:`` section and add module

View File

@ -7,16 +7,8 @@ Development Quickstart
.. _npm: https://www.npmjs.com/ .. _npm: https://www.npmjs.com/
.. _Node.js: https://nodejs.org/ .. _Node.js: https://nodejs.org/
SearXNG loves developers, just clone and start hacking. All the rest is done for
.. sidebar:: further read you simply by using :ref:`make <makefile>`.
- :ref:`makefile`
- :ref:`buildhosts`
SearXNG loves developers; Developers do not need to worry about tool chains, the
usual developer tasks can be comfortably executed via :ref:`make <makefile>`.
Don't hesitate, just clone SearXNG's sources and start hacking right now ..
.. code:: bash .. code:: bash
@ -26,23 +18,25 @@ Here is how a minimal workflow looks like:
1. *start* hacking 1. *start* hacking
2. *run* your code: :ref:`make run` 2. *run* your code: :ref:`make run`
3. *format & test* your code: :ref:`make format.python` and :ref:`make test` 3. *test* your code: :ref:`make test`
If you think at some point something fails, go back to *start*. Otherwise, If you think at some point something fails, go back to *start*. Otherwise,
choose a meaningful commit message and we are happy to receive your pull choose a meaningful commit message and we are happy to receive your pull
request. To not end in *wild west* we have some directives, please pay attention request. To not end in *wild west* we have some directives, please pay attention
to our ":ref:`how to contribute`" guideline. to our ":ref:`how to contribute`" guideline.
.. sidebar:: further read If you implement themes, you will need to setup a :ref:`make node.env` once:
- :ref:`make nvm` .. code:: bash
- :ref:`make themes`
If you implement themes, you will need to setup a :ref:`Node.js environment make node.env
<make node.env>`: ``make node.env``
Before you call *make run* (2.), you need to compile the modified styles and Before you call *make run* (2.), you need to compile the modified styles and
JavaScript: ``make themes.all`` JavaScript:
.. code:: bash
make themes.all
Alternatively you can also compile selective the theme you have modified, Alternatively you can also compile selective the theme you have modified,
e.g. the *simple* theme. e.g. the *simple* theme.
@ -55,10 +49,6 @@ e.g. the *simple* theme.
To get live builds while modifying CSS & JS use: ``LIVE_THEME=simple make run`` To get live builds while modifying CSS & JS use: ``LIVE_THEME=simple make run``
.. sidebar:: further read
- :ref:`make static.build`
If you finished your *tests* you can start to commit your changes. To separate If you finished your *tests* you can start to commit your changes. To separate
the modified source code from the build products first run: the modified source code from the build products first run:
@ -67,14 +57,14 @@ the modified source code from the build products first run:
make static.build.restore make static.build.restore
This will restore the old build products and only your changes of the code This will restore the old build products and only your changes of the code
remain in the working tree which can now be added & committed. When all sources remain in the working tree which can now be added & commited. When all sources
are committed, you can commit the build products simply by: are commited, you can commit the build products simply by:
.. code:: bash .. code:: bash
make static.build.commit make static.build.commit
Committing the build products should be the last step, just before you send us Commiting the build products should be the last step, just before you send us
your PR. There is also a make target to rewind this last build commit: your PR. There is also a make target to rewind this last build commit:
.. code:: bash .. code:: bash

View File

@ -37,7 +37,7 @@ docs.live <make docs.live>` to build HTML while editing.
- DOT_, `Graphviz's dot`_, Graphviz_ - DOT_, `Graphviz's dot`_, Graphviz_
.. contents:: .. contents:: Contents
:depth: 3 :depth: 3
:local: :local:
:backlinks: entry :backlinks: entry
@ -235,7 +235,7 @@ To refer anchors use the `ref role`_ markup:
.. admonition:: ``:ref:`` role .. admonition:: ``:ref:`` role
:class: rst-example :class: rst-example
Visit chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo Visist chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo
bar <reST anchor>`. bar <reST anchor>`.
.. _reST ordinary ref: .. _reST ordinary ref:
@ -292,7 +292,7 @@ content becomes smart.
files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst``` files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst```
pull request :pull:`4` ``:pull:`4``` pull request :pull:`4` ``:pull:`4```
patch :patch:`af2cae6` ``:patch:`af2cae6``` patch :patch:`af2cae6` ``:patch:`af2cae6```
PyPi package :pypi:`httpx` ``:pypi:`httpx``` PyPi package :pypi:`searx` ``:pypi:`searx```
manual page man :man:`bash` ``:man:`bash``` manual page man :man:`bash` ``:man:`bash```
intersphinx_ intersphinx_
-------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------

Some files were not shown because too many files have changed in this diff Show More