Commit Graph

45 Commits

Author SHA1 Message Date
Markus Heiser a9fc4885f2 [brand] SearXNG - bash env SEARXNG_URL
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Markus Heiser 60edf2623d [brand] SearXNG - reference /etc/searxng/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-11 12:28:55 +00:00
Alexandre Flament 253b850376 SearXNG: SEARXNG_SETTINGS_PATH 2021-10-02 17:18:05 +02:00
Markus Heiser b6a55e223c [mod] reduce enviroment variables in shell scripts to what is needed
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts.  This patch reduce the variables in ./utils/brand.env to what is
needed.  The following ('brand:' settings) can be removed from this file:

- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL

Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:

- GIT_URL            <--> brand.git_url
- GIT_BRANCH         <--> brand.git_branch
- SEARX_URL          <--> server.base_url  (aka PUBLIC_URL)
- SEARX_PORT         <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser f61c918dd4 [mod] normalize .config.sh with settings.yml
In commit 94851790 we have centralized all SearXNG setups in the settings.yml
file:

  94851790 [mod] move brand options from Makefile to settings.yml

This step has not yet been completed for the installation procedures!  Since all
SearXNG setups are done in the settings.yml these environment variables needs to
be removed from the ./conf.sh file.  Scripts and other tasks running outside of
an instance got the needed values from the ./utils/brand.env file.

By example: ATM the environment variables of the ./config.sh file are in
conflict with them from settings.yml:

  - PUBLIC_URL          --> {server:base_url}
  - SEARX_INTERNAL_HTTP --> {server:bind_address}.{server:port}
  - GIT_BRANCH          --> {brand:GIT_URL}

These environment variable of a SearXNG instance and additional

  - SEARX_SETTINGS_TEMPLATE

has been remove from the '.config.sh' file.  With this patch, the main focus of
./conf.sh resists on environment variables needed for the installation of morty,
filtron software.

modified  .config.sh:
  - removed no longer supported variables (see above)
  - add comment about: SearXNG setup in settings.yml

modified utils/searx.sh:
  - SEARX_INTERNAL_HTTP no longer take from .config.sh
  - SEARX_SETTINGS_PATH /etc/searx/settings.yml
  - SEARX_SETTINGS_TEMPLATE obsolete

modified utils/lib_install.sh:
  Initialize environment variables SEARX_PYENV, SEARX_SETTINGS_PATH and
  PUBLIC_URL.

modified:  utils/morty.sh
  Add missing hint about SEARX_SETTINGS_PATH and move PUBLIC_URL to
  utils/lib_install.sh

modified:  utils/morty.sh
  Move PUBLIC_URL to utils/lib_install.sh

Renamed utils/templates/etc/searx/use_default_settings.yml -> settings.yml
  - removed option which can't be modified after installation
  - add some comments with examples

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:57 +02:00
Markus Heiser 3e50e8de3e [mod] drop usage of the searx.brand namespace (python procs)
Added function searx.get_setting(name, default=_unset):
  Returns the value to which ``name`` point.  If there is no such name in the
  settings and the ``default`` is unset, a KeyError exception is raised.

In all the python processes ..

- make docs
- make buildenv
- make install (setup.py)

the usage of the 'brand.*' name space is replaced by 'searx.get_setting'
function.

- brand.SEARX_URL        --> get_setting('server.base_url')
- brand.GIT_URL          --> get_setting('brand.git_url')
- brand.GIT_BRANCH'      --> get_setting('server.base_url')
- brand.ISSUE_URL        --> get_setting('brand.issue_url')
- brand.DOCS_URL         --> get_setting('brand.docs_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.CONTACT_URL      --> get_setting('general.contact_url', '')
- brand.WIKI_URL         --> get_setting('brand.wiki_url')
- brand.TWITTER_URL      --> get_setting('brand.twitter_url', '')

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-18 15:55:42 +02:00
Markus Heiser 4a814dabf3 [yamllint] ./utils/templates/etc/searx/*.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-12 20:46:07 +02:00
Markus Heiser 6ed4616da9 [enh] add settings option to enable/disable search formats
Access to formats can be denied by settings configuration::

    search:
        formats: [html, csv, json, rss]

Closes: https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 08:32:52 +02:00
Alex Balgavy 8736f5bd70 Use $host in nginx morty.conf template 2021-03-04 11:16:27 +01:00
Alex Balgavy 6b59800dc6 Fix security vulnerabilities in suggested nginx configuration
The suggested configurations for nginx found in the documentation and
templates lead to vulnerabilities allowing host spoofing [1] and path
traversal [2], as reported by Gixy [3]. This commit fixes those issues.

[1] https://github.com/yandex/gixy/blob/master/docs/en/plugins/hostspoofing.md
[2] https://github.com/yandex/gixy/blob/master/docs/en/plugins/aliastraversal.md
[3] https://github.com/yandex/gixy
2021-03-03 12:34:22 +01:00
Alexandre Flament 6e2872f436 [enh] add searx.shared
shared dictionary between the workers (UWSGI or werkzeug)
scheduler: run a task once every x seconds (UWSGI or werkzeug)
2021-01-12 11:47:17 +01:00
Markus Heiser a70b9b9f61 [doc] recommend to use 'use_default_settings=True'
Since #2291 is merged, it is recommend to use::

  use_default_settings=True

1. Add a template file use_default_settings.yml::

    SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"

2. In Chapter "Configuration" recommend to make use of
   'use_default_settings=True' and describe it

3. Rewrite of docs/admin/settings.rst
   - move chapter 'settings.yml location' to the top
   - update and split chapter 'Global Settings'

4. Add environment SEARX_SETTINGS_TEMPLATE to .config.sh

5. Use environment $SEARX_SETTINGS_TEMPLATE in the utils/searx.sh script

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-12-08 20:19:10 +01:00
renyhp b386a815da
Fix typo chmod searx:searx > chown searx:searx 2020-10-19 17:31:02 +02:00
Markus Heiser f9faafa896
[fix] external_bang - UnicodeDecodeError: 'ascii' codec can't decode (#2043)
Python's default encoding depends on the platform, set (python) default encoding
UTF-8 in uwsgi ini files:

    LANG=C.UTF-8
    LANGUAGE=C.UTF-8
    LC_ALL=C.UTF-8

Error pattern:

    Traceback (most recent call last):
      File "/usr/local/searx/searx-src/searx/webapp.py", line 74, in <module>
        from searx.search import SearchWithPlugins, get_search_query_from_webapp
      File "/usr/local/searx/searx-src/searx/search.py", line 32, in <module>
        from searx.external_bang import get_bang_url
      File "/usr/local/searx/searx-src/searx/external_bang.py", line 13, in <module>
        for bang in json.load(json_file)['bang']:
      File "/usr/lib/python3.8/json/__init__.py", line 293, in load
        return loads(fp.read(),
      File "/usr/lib/python3.8/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
      UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 31341: ordinal not in range(128)

close: https://github.com/asciimoo/searx/issues/2041

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-07-11 12:17:06 +02:00
Markus Heiser 6ff20cef73 [fix] indentation of filtron's rules (json)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-06-18 18:31:46 +02:00
Markus Heiser ea3255835a utils/morty.sh: set morty key to avoid service abuse
- https://github.com/asciimoo/searx/issues/1871#issuecomment-592459798

    make install all

generates random MORTY_KEY, install service with that key and sets option in the
searx settingy.yml file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-28 16:21:45 +02:00
Markus Heiser 58d5da8b57 nginx: normalize installation (docs and script)s over all distros
This is the revision of the documentation about the varous nginx installation
variants.  It also implements the nginx installation scripts for morty and
filtron.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-11 13:19:11 +02:00
Markus Heiser f693149cde Changes from the installation tests on (all) LXC containers.
Tested and fixed HTTP & uWSGI installation on:

  ubu1604 ubu1804 ubu1910 ubu2004 fedora31 archlinux

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-08 18:38:36 +02:00
Markus Heiser ee39a098ac apache: normalize installation (docs and script)s over all distros
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-07 18:31:51 +02:00
Markus Heiser eb0d4646d8 docs: rework of chapter "Install with apache"
BTW: normalize installation-nginx.rst
2020-04-06 17:59:06 +02:00
Markus Heiser c81849cb5a filtron.sh & morty.sh: improve usage message (if used in containers)
BTW: normalize soma variable names

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-05 17:40:37 +02:00
Markus Heiser e530e20ae6 misc: fix variuous marginals
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-04-04 17:53:16 +02:00
Markus Heiser 7b4cf2eb48 tooling box: simplify build enviroments
- no more need for a .config.mk
- docs: use searx.brands environment
- searx.sh, filtron.sh & morty.sh are sourcing utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-29 15:09:34 +02:00
Markus Heiser c15337850e fix: minor typos
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-06 22:06:19 +01:00
Markus Heiser 387c6a7769 docs: improve description of uwsgi & ngingx setup
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-03-06 14:47:00 +01:00
Markus Heiser af6acd3417 LXC: install searx-suite installs searx, filtron & morty on all containers
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-26 19:07:55 +01:00
Markus Heiser d5917cc029 utils/lib.sh: make uWSGI installation available for all distros
support: ubuntu, debin, fedora, archlinux

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-25 20:20:17 +01:00
Markus Heiser 59e4026762 searx.sh: install settings at /etc/searx/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-17 18:58:59 +01:00
Markus Heiser de58f02f6b filtron: add missing log action to the filtron rules
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-17 15:36:10 +01:00
Markus Heiser 0d6153db12 filtron.sh: updated rules from production
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-11 15:57:42 +01:00
Markus Heiser ed4cb4f160 tooling box: varius fix from tests 2020-02-08 13:24:08 +01:00
Markus Heiser 71d7550dbe tooling box ./utils/*: minor fix from production test 2020-02-04 19:47:33 +01:00
Markus Heiser 2f40f61f83 /etc/filtron/rules.json: normalize rules from docs & tooling box
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-04 17:59:58 +01:00
Markus Heiser a4437c47ac utils/morty.sh: add script to install morty result proxy
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-02-02 18:14:10 +01:00
Markus Heiser 0bb8847087 utils/filtron.sh: add option to debug filtron requests
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-31 17:25:38 +01:00
Markus Heiser 91a55e159e apache: reverse proxy, set `ProxyPreserveHost On`
related discussions:

- https://github.com/asciimoo/searx/issues/1822
- https://github.com/asciimoo/searx/issues/1819#issuecomment-580400259

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-31 15:54:07 +01:00
Markus Heiser 6274a54746 utils/searx.sh & filtron.sh: misc changes from first tests (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-30 19:55:51 +01:00
Markus Heiser 924bf65517 utils/searx.sh & filtron.sh: misc changes from first tests (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-29 20:00:50 +01:00
Markus Heiser 971a8264b2 utils/searx.sh: add apache site searx.conf:uwsgi (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-27 19:08:40 +01:00
Markus Heiser d171fcd56e utils/searx.sh: add apache site searx.conf:uwsgi
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-21 18:38:57 +01:00
Markus Heiser 9b5a7f7559 utils/searx.sh: add script to install isolated searx service
First version which serves searx over uwsgi at http://127.0.0.1:8888

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-20 16:55:05 +01:00
Markus Heiser 89df9d9141 utils/searx.sh: add script to install isolated searx service (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-16 14:01:38 +01:00
Markus Heiser b5449ec47c filtron: log suspiciously frequent queries (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-13 18:37:05 +01:00
Markus Heiser 4990b07b4b utils/filtron.sh: various fix from first installation test (WIP)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-09 16:25:05 +01:00
Markus Heiser 4139c63d23 utils/filtron.sh: add script to install filtron middleware
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2020-01-08 19:21:07 +01:00