jandieman
/
searxng
mirror of https://github.com/searxng/searxng.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
8,220 Commits 6 Branches 17 Tags 134 MiB
Commit Graph

390 Commits

Author SHA1 Message Date
Markus Heiser 809bf85426 [perf] pyenvinstall: reinstall only when requirements are change 
Create a (PY_ENV)/requirements.sha256 file with sha256 sums of the
./requirements*.txt files and reinstall only if the sha256 sums change.

Suggested-by: @dalf / https://github.com/searx/searx/pull/2418#pullrequestreview-558426579
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-24 16:25:09 +01:00
Markus Heiser d85012b49b [perf] virtualenv: install boilerplate only once 
Install boilerplate packages (pip, wheel, setuptools and requirements.txt) only
once, when virtualenv is created.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-24 15:59:57 +01:00
Markus Heiser aace837665 [fix] makefile.sphinx: additions to commit c2a6f145 
In commit c2a6f145 the sphinx-doc and sphinx-live targets has been removed,
since Sphinx is already installed by the pyenvinstall target.

This commit removes the targets from the remaining prerequisites of the 'BOOK'
targets

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-24 13:54:07 +01:00
Alexandre Flament 2fc2d17a27
Merge pull request #2418 from return42/fix-pyenv-install 
[fix] makefile.python: remove duplicate pyenv-(un)install targets
 2020-12-24 10:23:09 +01:00
Markus Heiser af49bd2bd2 [fix] makefile.python: remove duplicate pyenv-(un)install targets 
Makefile targets 'pyenv-install' and 'pyenv-uninstall' are unused duplicates of
'pyenvinstall' and 'pyenvuninstall'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-24 09:18:22 +01:00
Markus Heiser 5fc9273e23 [mod] utils/searx.sh install: create branch if not already exists 
This patch keeps an eye on:

1. When installing searx, the branch to install needs to be a local branch.
2. The branch to install can be different from the current branch.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-23 16:06:06 +01:00
Markus Heiser 0fc3759d74 [fix] use SEARX_SETTINGS_TEMPLATE from .config environment 
In commit a70b9b9f the SEARX_SETTINGS_TEMPLATE environment was added to the
.config file, but was not use in utils/searx.sh.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-23 14:10:07 +01:00
Alexandre Flament db5b060455 [fix] update buildenv 
CONTACT_URL is unset in Makefile, but searx/brand.py and
utils/brand.env are not updated.

This commit fixes this issue.
 2020-12-21 10:55:28 +01:00
Alexandre Flament 72d2bb2289
Merge pull request #2399 from return42/fix-lxc 
[fix] determine path to makefile.lxc in a LXC
 2020-12-20 09:32:00 +01:00
Markus Heiser be08e5bfab [fix] yum installation of 'Development Tools' 
'yum install' does not support '@development-tools' notation for group
installation.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-18 22:32:05 +01:00
Markus Heiser 1a4524e03c [mod] remove obsolete virtualenv command 
Installation and use of the command 'virtualenv' was only needed in py2 and py2
is no longer suported by searx.  In py3 the command is replaced by 'python -m
venv'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-18 22:31:13 +01:00
Markus Heiser c89df6739f [fix] determine path to makefile.lxc in a LXC 
The path to the makefile.lxc is not always static and
'/share/searx/utils/makefile.lxc' can be wrong.

When target 'lxc.activate' is called within the LXC container the path to
makefile.lxc depends on LXC_SHARE_FOLDER and the basename of the folder where
searx was cloned inital on the HOST system.

This patch determines the path of 'makefile.lxc' next to 'makefile.include'
itself.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-18 17:10:49 +01:00
BBaoVanC 19fce74443
Add link to contact instance maintainer to footer of each page (#2391) 2020-12-18 09:53:28 +01:00
Alexandre Flament 7ec8bc3ea7 [mod] split searx.search into different processors 
see searx.search.processors.abstract.EngineProcessor

First the method searx call the get_params method.

If the return value is not None, then the searx call the method search.
 2020-12-17 11:39:36 +01:00
Markus Heiser a70b9b9f61 [doc] recommend to use 'use_default_settings=True' 
Since #2291 is merged, it is recommend to use::

  use_default_settings=True

1. Add a template file use_default_settings.yml::

    SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"

2. In Chapter "Configuration" recommend to make use of
   'use_default_settings=True' and describe it

3. Rewrite of docs/admin/settings.rst
   - move chapter 'settings.yml location' to the top
   - update and split chapter 'Global Settings'

4. Add environment SEARX_SETTINGS_TEMPLATE to .config.sh

5. Use environment $SEARX_SETTINGS_TEMPLATE in the utils/searx.sh script

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-12-08 20:19:10 +01:00
Alexandre Flament 38d32337e7 [mod] documentations & comments: update http://* URL to https://*. 
About http://lesscss.org see https://github.com/less/less-docs/issues/520
 2020-12-04 16:52:25 +01:00
rachmadani haryono c03e4c86bc
Feature/standalone searx update (#1591) 
* chg: dev: update standalone_searx

parent d8a5df721b33dd8a7cc9e21dba4060f21d629f69
author rachmadaniHaryono <foreturiga@gmail.com> 1603896594 +0800
committer rachmadaniHaryono <foreturiga@gmail.com> 1603896619 +0800

chg: dev: debug engine_shortcuts
chg: dev: only initilize if engine is given
chg: dev: split main
chg: dev: standalone_searx
chg: dev: update standalone_searx
chg: doc: remove unnecessary log
chg: test: differentiate travis
chg: test: disable shortcut
chg: test: use default engine settings
fix: dev: category choices
fix: dev: duplicate engine shortcut
fix: dev: travis python3
fix: test:  use empty string as shortcut
fix: test: apkm
fix: test: engine shortcut
fix: test: mypy
fix: test: parameter
fix: test: pep8
fix: test: py2 compatibilities
fix: test: searx settings
fix: test: travis engines
new: dev: deduplicate engine
new: dev: main receive engines parameter
new: dev: parse_argument accept engines parameter
new: dev: split search query from get_result func
new: test: basic result case
Suggestions: use RawTextQuery to make the suggestions URLs. Update all themes accordingly.

* new: doc: searx import and init

* chg: dev: parse_argument

- doc
- run on __main__
- simple parse_args

* chg: doc: module

* chg: dev: import section

- remove unused python path modification
- new required package

* chg: dev: script run

- parse_argument func return directly parsed results
- main func return dict instead json text
- dump directly on sys.stdout.write

* chg: dev: get_search_query and get_search_query func

* chg: dev: main func

- move inner function outside
- return dict instead of json text

* new: dev: add utils to doc sys path

* new: doc: standalone_searx

* fix: doc: run script

* chg: dev: mypy type hint

* chg: dev: SearchQuery don't have attr engines

* chg: dev: reset engines __init__

* chg: test: unit test update

* chg: dev: pylint and flake8

* new: test: standalone_searx

* chg: dev: main func and doc

* chg: dev: import and type hint

* new: dev: main func

- remove get_result func
- single func which just translate dict

* chg: test: put mypy on dev requirement

* chg: doc: update

* new: doc: add standalone_searx module member

* chg: doc: shell command line

* chg: dev: remove mypy

* chg: doc: module docstring
 2020-11-04 12:38:54 +00:00
Markus Heiser e697d78d5a [mod] make docs-live - remove --port option from sphinx-autobuild 
Setting port fixed to 8080 has drawbacks when this port is already in use from
another application [1].  Removing this option from the command line offers the
use of the sphinx-autobuild defaults which are much more flexible [2].

[1] https://github.com/searx/searx/issues/2282
[2] https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-10-30 20:30:20 +01:00
Alexandre Flament 54afe37e18 [mod] .travis.yml: build documentation automatically on the master branch 2020-10-28 22:48:16 +01:00
Alexandre Flament ed6696e6bf [mod] add external_urls.json and wikidata_units.json 2020-10-28 08:09:25 +01:00
Marc Abonce Seguin 32957cdf49 add Ahmia filter plugin for onion results 2020-10-25 17:59:43 -07:00
renyhp b386a815da
Fix typo chmod searx:searx > chown searx:searx 2020-10-19 17:31:02 +02:00
Alexandre Flament f204e4903d [fix] migration from github.com/asciimoo/searx to github.com/searx/searx : fix URLs 2020-09-28 16:44:14 +02:00
Adam Tauber 8d47142f35
Merge pull request #2189 from dalf/architecture-clean-up 
Architecture clean up
 2020-09-28 14:56:23 +02:00
Alexandre Flament c2a6f14516 [fix] Update Sphinx package versions 
* Update call to sphinx-autobuild (remove --poll -B options, add --open-browser)
* Upgrade the Sphinx and sphinx-tabs versions.
* utils/makefile.sphinx: remove the SPHINXVERS variable.
* utils/makefile.sphinx: remove the sphinx-doc and sphinx-live targets, Sphinx is already installed by the pyenvinstall target.
 2020-09-28 13:27:50 +02:00
Alexandre Flament 485a502b88 [mod] add typing and __slots__ 2020-09-22 19:05:05 +02:00
Alexandre Flament 691d12726b [mod] check the engine tokens in searx/webadapter.py instead of searx/search.py 2020-09-22 18:59:51 +02:00
Alexandre Flament 2dbc0de0cd [mod] add searx/webadapter.py 
* move searx.search.get_search_query_from_webapp to searx.webadapter
* move searx.query.SearchQuery to searx.search
 2020-09-22 18:59:51 +02:00
Marc Abonce Seguin c86504b47a fix utils/fetch_languages to work with new languages 2020-09-22 11:37:44 +02:00
Markus Heiser 69a61ce333 [fix] remove doc-build for centos-7 from './utils/searx.sh doc' 
Since there is no centos-7 documentation 'searx.sh doc' produce an error
message::

    ERROR: centos-: uWSGI not yet implemented

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-09-20 11:11:14 +02:00
Dalf 1022228d95 Drop Python 2 (1/n): remove unicode string and url_utils 2020-09-10 10:39:04 +02:00
Markus Heiser 623fb87065
[enh] add centos-7 to LXC images (#2118) 
## What does this PR do?

To test centos-7 on non centos systems add centos-7 image to the LXC suite.

## Why is this change important?

PR #2112 adds centos-7 support to the install scripts

## How to test this PR locally?

`sudo ./utils/lxc.sh build searx-centos7`
 2020-08-08 19:50:11 +02:00
Eliesemoule 1986b5ecac
Utility scripts adapted to run on CentOS 7 (#2112) 2020-08-01 22:12:44 +02:00
Markus Heiser f9faafa896
[fix] external_bang - UnicodeDecodeError: 'ascii' codec can't decode (#2043) 
Python's default encoding depends on the platform, set (python) default encoding
UTF-8 in uwsgi ini files:

    LANG=C.UTF-8
    LANGUAGE=C.UTF-8
    LC_ALL=C.UTF-8

Error pattern:

    Traceback (most recent call last):
      File "/usr/local/searx/searx-src/searx/webapp.py", line 74, in <module>
        from searx.search import SearchWithPlugins, get_search_query_from_webapp
      File "/usr/local/searx/searx-src/searx/search.py", line 32, in <module>
        from searx.external_bang import get_bang_url
      File "/usr/local/searx/searx-src/searx/external_bang.py", line 13, in <module>
        for bang in json.load(json_file)['bang']:
      File "/usr/lib/python3.8/json/__init__.py", line 293, in load
        return loads(fp.read(),
      File "/usr/lib/python3.8/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
      UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 31341: ordinal not in range(128)

close: https://github.com/asciimoo/searx/issues/2041

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-07-11 12:17:06 +02:00
Markus Heiser 8d318ee142
Merge branch 'master' into gigablast 2020-06-29 16:09:59 +00:00
Markus Heiser 18805ee352 [fix] markup of auto generated installation instruction (uWSGI part) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-06-24 15:15:15 +02:00
Markus Heiser 4eab5cf854
Merge branch 'master' into gigablast 2020-06-19 04:51:48 +00:00
Markus Heiser 15924bcb67 [fix] buildprocess of the docs by adding sphinx-doc-prebuilds target 
The sphinx-doc-prebuilds is the single point of definition for prerequisites
each doc build needs

BTW: remove $(LXC_ENV_FOLDER) offset from BOOKS_FOLDER and DOCS_FOLDER path.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-06-18 18:44:22 +02:00
Markus Heiser 51428de0fe [fix] add missing latexmk package to debian's BUILD_PACKAGES 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-06-18 18:35:59 +02:00
Markus Heiser 6ff20cef73 [fix] indentation of filtron's rules (json) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-06-18 18:31:46 +02:00
Markus Heiser 1510382558 utils/fetch_languages.py: print more meaningfull messages 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-05-28 15:18:47 +02:00
Markus Heiser f576cadfba [fix] installation instructions: pip install -e searx (setup.py) 
In the past we did not really install searx into the virtualenv.  This is a
problem, since entry-points and other python installation stuff is not trigger.

See discussion: https://github.com/asciimoo/searx/pull/1938#issuecomment-632056508

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-05-21 15:46:43 +02:00
Markus Heiser e50ccc99c0 archlinux: add package 'inetutils' to boilerplate 
Even it is a core component, some hoster do not have pre installed the
'inetutils' package.  We do need this package for the 'hostname' command.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-05-04 16:15:23 +02:00
Markus Heiser 0973255aff [fix] fatal: repository '/root/searx' does not exist 
The $SERVICE_USER (searx) needs read permission to clone repository.  The
installation script stops if permission is not granted.

- https://github.com/asciimoo/searx/pull/1803#issuecomment-616406528

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-30 17:49:26 +02:00
Markus Heiser 8829796f76 utils/morty.sh option new-key: set new morty key 
A command to set new morty key in morty & searx service (settings.yml) and to
restart both services with new key::

  utils/morty.sh option new-key

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-28 16:54:42 +02:00
Markus Heiser ea3255835a utils/morty.sh: set morty key to avoid service abuse 
- https://github.com/asciimoo/searx/issues/1871#issuecomment-592459798

    make install all

generates random MORTY_KEY, install service with that key and sets option in the
searx settingy.yml file.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-28 16:21:45 +02:00
Markus Heiser 9504acaca6 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-04-23 12:49:56 +02:00
Markus Heiser d21e0b0ef8 [fix] old images need an initial 'apt update' 
see https://github.com/asciimoo/searx/pull/1803#issuecomment-616684353

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-22 01:11:25 +02:00
Markus Heiser 697dd6c6d3 [fix] ffox useragent: use Windows NT 10.0 and not Windows NT 10 
This is an addittion to PR #1934:

  The .0 change really counts on some engines like Bing which seems to allow
  Windows NT 10.0 but not Windows NT 10.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-22 00:43:42 +02:00
Markus Heiser 99ff16c465 tooling box: added nginx + polished bash scripts and environment 
- add installation method for nginx sites, morty and filtron
- clean up PUBLIC_URL environment in and outside of containers
- clean up comand lines
- handle uWSGI quirks on fedora (emperor mode)
- handle Python quirks on debian (there is no 'python' command anymore)
- lib.sh: add die and die_caller functions
- lxc_suite_install_info is now a function
- lint: shellcheck

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-13 11:34:28 +02:00
Markus Heiser 58d5da8b57 nginx: normalize installation (docs and script)s over all distros 
This is the revision of the documentation about the varous nginx installation
variants.  It also implements the nginx installation scripts for morty and
filtron.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-11 13:19:11 +02:00
Markus Heiser 67afa6b1fb bash scripts: source code cleaned up (shellcheck) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-10 14:00:52 +02:00
Markus Heiser 44edc92569 [fix] morty: PUBLIC_URL_PATH_MORTY needs trailing slash 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-09 14:28:34 +02:00
Markus Heiser f693149cde Changes from the installation tests on (all) LXC containers. 
Tested and fixed HTTP & uWSGI installation on:

  ubu1604 ubu1804 ubu1910 ubu2004 fedora31 archlinux

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-08 18:38:36 +02:00
Markus Heiser ee39a098ac apache: normalize installation (docs and script)s over all distros 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-07 18:31:51 +02:00
Markus Heiser eb0d4646d8 docs: rework of chapter "Install with apache" 
BTW: normalize installation-nginx.rst
 2020-04-06 17:59:06 +02:00
Markus Heiser c81849cb5a filtron.sh & morty.sh: improve usage message (if used in containers) 
BTW: normalize soma variable names

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-05 17:40:37 +02:00
Markus Heiser c2caf9569c [fix] creation of /.lxcenv in containers 
In utils/lib.sh there are two functions with the same name, but different tasks.
Rename one of them from lxc_init_container() into lxc_init_container_env().

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-05 15:15:28 +02:00
Markus Heiser c748047e92 utils/searx.sh: add cmd to (un)set option image_proxy and result_proxy 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-04 17:54:20 +02:00
Markus Heiser e530e20ae6 misc: fix variuous marginals 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-04 17:53:16 +02:00
Markus Heiser 3c19f2f1ce utils/lxc.sh install: support installation on selected containers 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-04 12:40:14 +02:00
Markus Heiser e32b08f616 utils/morty.sh: add generic documentation 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-03 20:26:20 +02:00
Markus Heiser 853be4e11c utils/filtron.sh: add generic documentation 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-03 20:24:40 +02:00
Markus Heiser af988dbf71 utils/lxc.sh: support build of selected containers 
$ sudo -H ./utils/lxc.sh build  <name>
  $ sudo -H ./utils/lxc.sh show   [images|suite|info|config [<name>]]

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-03 17:08:42 +02:00
Markus Heiser f32b4fcedd LXC: add virtualenv to LXC_BASE_PACKAGES 
Commit 09a40625 adds virtualenv dependency.  BTW remove deprecated
--no-site-packages.  Not having access to global site-packages is now the
default behavior.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-02 20:27:44 +02:00
Markus Heiser 2441e24288 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-04-02 18:59:43 +02:00
Markus Heiser 09a4062505 makefile.python: partial revert off 9b48ae47, adds py2 support back 
revert "makefile.python: ... drop build support py2" to get back Py2 support.
TPy2 support need virtualenv installed by the OS.

BTW: log environment and python version in travis's install phase

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-01 17:21:26 +02:00
Markus Heiser 31437d04e8 makefile.python: hardening - start pylint by module 
- replace: $(PY_ENV_BIN)/pylint --> $(PY_ENV_BIN)/python -m pylint

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-01 16:07:59 +02:00
Markus Heiser 9b48ae47ec makefile.python: hardening - drop virtualenv, drop build support py2 
From py3.3 on a subset of virtualenv is built-in, so you can run '-m venv' ot of
the box.

- replace: $(PY_ENV_BIN)/pip --> $(PY_ENV_BIN)/python -m pip
- remove obsolete virtualenv-exe target and adjust VTENV_OPTS
- remove obsolete msg-pip-exe target
- print list of py launchers available from $(PY_ENV_BIN) to the log
- fix hard coded ./local

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-01 14:16:49 +02:00
Markus Heiser 3bbcfe6ab3 travis: make travis build more verbose 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-01 11:57:48 +02:00
Markus Heiser 6007bd7fc8 makefile.python: fix (old) typo 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-04-01 10:59:51 +02:00
Markus Heiser f27f6c6c5d docs: use make and uitls/searx.sh in containers 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-31 18:25:40 +02:00
Markus Heiser dd53c45a2c docs: add utils/lxc.sh docs, normalize filtron, morty & searx docs 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-30 18:47:01 +02:00
Markus Heiser 3e4d022d04 [fix] make test.sh: fix various shellcheck error messages 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-30 11:03:46 +02:00
Markus Heiser 5e2b161379 Makefile: marginal fix 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-29 16:46:07 +02:00
Markus Heiser 0742c2504d build environment: add GIT_BRANCH variable 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-29 16:28:17 +02:00
Markus Heiser 7b4cf2eb48 tooling box: simplify build enviroments 
- no more need for a .config.mk
- docs: use searx.brands environment
- searx.sh, filtron.sh & morty.sh are sourcing utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-29 15:09:34 +02:00
Markus Heiser 65b22be1f4 Merge PR #1900 of https://github.com/asciimoo/searx into filtron 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-29 13:03:27 +02:00
Markus Heiser 442b27ac1b bash: use build environment from Makefile 
To Makefile target brand, add creation of bash environment in::

  utils/brand.env

In bash scripts (manage.sh) source env by::

  . utils/brand.env

manage.sh help: show GIT_URL and more environment

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-29 11:41:59 +02:00
Markus Heiser 1fcec0bbda lxc.sh install base: add bash and python3 to LXC_BASE_PACKAGES 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-23 18:06:35 +01:00
Markus Heiser baf1ffd1fa LXC: set LXC_ENV_FOLDER=lxc/$(shell hostname)/ 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-23 18:05:46 +01:00
Markus Heiser 21ba25ebb8 Merge branch 'filtron' of https://github.com/return42/searx into filtron 2020-03-23 15:02:32 +01:00
Markus Heiser 2d956696e9 docs: building (PDF) books / build user book 
BTW: cleaned up Makefile target help

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-21 18:45:38 +01:00
Markus Heiser 62da769952 ./utils/lxc.sh: add command 'install base' 
to install basic packages like git and *build essentials*::

  $ ./utils/lxc.sh install base

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-18 17:47:48 +01:00
Markus Heiser d2cfe9ce5b LXC: add /.lxcenv 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-18 15:34:46 +01:00
Markus Heiser 86e79488aa LXC: utils/makefile.lxc (inital) add /.lxcenv.mk to contaiiners 
Get LXC environment when building make targets.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-15 17:01:36 +01:00
Markus Heiser 80f7b65804 searx.sh: add buildhost cmd, installs OS packages for builds 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-09 01:37:26 +01:00
Markus Heiser f5f83b8a99 shellcheck: minor fixes to support all dsitros 
checked with::

  sudo ./utils/lxc.sh cmd -- make test.sh

on ubu1804, ubu1910, fedora31 and archlinux.  All checks OK except:

ubu1604: uses shellcheck v0.3.7 (from 04/2015) which is no longer supported!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-08 18:37:16 +01:00
Markus Heiser 04ad648105 makefile.python: remove python2 support 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-08 18:35:38 +01:00
Markus Heiser 938bb02550 searx.sh: add commandline 'install packages' 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-08 18:32:30 +01:00
Markus Heiser d48c7bf678 LXC: fixed raw.idmap & removed obsolete uid/gid subordinate 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-08 18:30:25 +01:00
Markus Heiser e7f69b63f1 LXC: from searx.sh, morty.sh and filtron.sh tests 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-08 02:41:45 +01:00
Markus Heiser b1e90cff23 LXC: separate lxc-suite from lxc & improved command line. 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-07 20:24:08 +01:00
Markus Heiser c15337850e fix: minor typos 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-06 22:06:19 +01:00
Markus Heiser 387c6a7769 docs: improve description of uwsgi & ngingx setup 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-06 14:47:00 +01:00
Markus Heiser cbc08fdc26 docs: describe uwsgi setup of all suported distributions 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-04 19:56:40 +01:00
Markus Heiser 3aff2c19d1 docs: complement uwsgi documentation for all distros 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-04 12:22:20 +01:00
Markus Heiser 9f1cb7a943 Merge branch 'master' of https://github.com/asciimoo/searx into filtron 2020-03-04 12:11:58 +01:00
Markus Heiser 3cb7daedb7 docs: generic doocumentation for searx-uwsgi 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-03 19:57:15 +01:00
Markus Heiser f25da6070e docs: generic documentation get templating 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-03 16:26:02 +01:00
Markus Heiser 7af991acda LXC: add ubuntu 19.10 and 20.04 to the list of test images 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-02 19:01:08 +01:00
Markus Heiser c3e4753ce9 docs: generic documentation from the installation scripts 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-02 19:00:19 +01:00
Markus Heiser 491cb95a1f utils/lxc.env: separate environment that is used in containers 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-01 18:28:10 +01:00
Markus Heiser 40843fe95a searx/data/engines_languages.json: sort json file (no content change) 
To get meaningfull diffs, the json file has to be sorted.  Before applying any
further content patch, the json file needs a inital sort (without changing any
content).

Sorted by::

  import sys, json

  with open('engines_languages.json') as f:
      j = json.load(f)

  with open('engines_languages.json', 'w') as f:
      json.dump(j, f, indent=2, sort_keys=True)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-01 07:56:46 +01:00
Markus Heiser 37c135f2ce LXC: improved UX when working with a bunch of containers 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-27 19:13:03 +01:00
Markus Heiser af6acd3417 LXC: install searx-suite installs searx, filtron & morty on all containers 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-26 19:07:55 +01:00
Markus Heiser d5917cc029 utils/lib.sh: make uWSGI installation available for all distros 
support: ubuntu, debin, fedora, archlinux

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-25 20:20:17 +01:00
Markus Heiser f17b28743a utils/serx.sh: make uWSGI installation available for all distros (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-23 21:05:22 +01:00
Markus Heiser 5fb6d4f508 LXC: normalize package installation & user creation. 
utils/lib.sh:
- get DIST_ID & DIST_VERSION from /etc/os-release
- pkg_[install|remove|...] supports ubuntu, debian, archlinux & fedora

utils/lxc.sh
- Workaround for the "setrlimit(RLIMIT_CORE): Operation not permitted" error::
    'Set disable_coredump false' >> /etc/sudo.conf

utils/[searx.sh|filtron.sh|morty.sh]
- switched user creation from 'adduser' perl script to 'useradd' built-in
  command

utils/searx.sh
- install packages for ubuntu, debian, archlinux & fedora

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-23 12:10:45 +01:00
Markus Heiser e36e0f80ae LXC: added archlinux and fedora31 containers 
BTW: update all packages when installing conatiner's boilerplate

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-18 18:40:34 +01:00
Markus Heiser ccb96544f7 LXC: improved console messages & prompt 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-18 18:20:03 +01:00
Markus Heiser 59e4026762 searx.sh: install settings at /etc/searx/settings.yml 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-17 18:58:59 +01:00
Markus Heiser de58f02f6b filtron: add missing log action to the filtron rules 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-17 15:36:10 +01:00
Markus Heiser 0bfc61dbe9 tooling box: misc fixes from lxc tests 
Tested by:

  sudo -H ./utils/lxc.sh build containers
  sudo -H ./utils/lxc.sh cmd /share/searx/utils/searx.sh    install all
  sudo -H ./utils/lxc.sh cmd /share/searx/utils/filtron.sh  install all
  sudo -H ./utils/lxc.sh cmd /share/searx/utils/morty.sh    install all

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-16 22:29:06 +01:00
Markus Heiser a1f5f2ced8 LXC: minor fixes and renaming 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-16 22:26:03 +01:00
Markus Heiser ad32739860 shellcheck: fix usse -n instead of ! -z (SC2236 SC2237) 
- https://www.shellcheck.net/wiki/SC2236 -- Use -n instead of ! -z.
 - https://www.shellcheck.net/wiki/SC2237 -- Use [ -n .. ] instead of ! [ -z ....

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-16 20:07:37 +01:00
Markus Heiser e8cf225046 LXC: tooling box - add boilerplate to containers 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-16 18:18:15 +01:00
Markus Heiser 0b2b325552 LXC: tooling box - add HOST shares and commandline to containers (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-13 18:25:03 +01:00
Markus Heiser e6bf2038c3 filtron.sh: add 'install rules' command 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-11 15:58:35 +01:00
Markus Heiser 0d6153db12 filtron.sh: updated rules from production 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-11 15:57:42 +01:00
Markus Heiser 7751b29559 LXC: add LXC tooling box (initial, WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-08 19:12:28 +01:00
Markus Heiser 5c0f9c8c79 tooling box: varius fix from tests 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-08 17:12:01 +01:00
Markus Heiser ed4cb4f160 tooling box: varius fix from tests 2020-02-08 13:24:08 +01:00
Markus Heiser a014fa0ccd upload-pypi: twine is needed to upload to PyPi 'make upload-pypi' 
To build & deploy packages on PyPi call make target:

  make upload-pypi

- https://github.com/asciimoo/searx/issues/1829

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-05 15:26:57 +01:00
Markus Heiser 71d7550dbe tooling box ./utils/*: minor fix from production test 2020-02-04 19:47:33 +01:00
Markus Heiser 2f40f61f83 /etc/filtron/rules.json: normalize rules from docs & tooling box 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-04 17:59:58 +01:00
Markus Heiser eedd63ccd5 docs: revision of the installation instructions 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-04 16:42:13 +01:00
Markus Heiser 79e1f837ef utils & Makefile: add .config.mk & .config.sh for searx brands 
By isolating the environment of makefiles and bash scripts into .config.mk and
.config.sh it is simple to maintain searx brands by setting some central
environments.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-04 10:39:42 +01:00
Markus Heiser 94ac560dcc doc: add descriptions of our tooling box in ./utils 
Initial started with the scripts for searx, filtron and morty installations.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-03 13:25:51 +01:00
Markus Heiser e7401796f6 utils/morty.sh: fixed public url 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-02 18:20:20 +01:00
Markus Heiser a4437c47ac utils/morty.sh: add script to install morty result proxy 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-02 18:14:10 +01:00
Markus Heiser 709ac51d33 utils/filtron.sh: generalize systemd, accounts and golang tasks 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-01 16:59:27 +01:00
Markus Heiser 56a93ee770 utils/lib.sh: colorize messages 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-31 18:24:31 +01:00
Markus Heiser 0bb8847087 utils/filtron.sh: add option to debug filtron requests 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-31 17:25:38 +01:00
Markus Heiser 91a55e159e apache: reverse proxy, set `ProxyPreserveHost On` 
related discussions:

- https://github.com/asciimoo/searx/issues/1822
- https://github.com/asciimoo/searx/issues/1819#issuecomment-580400259

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-31 15:54:07 +01:00
Markus Heiser cd9b877c8c utils/searx.sh & filtron.sh: misc changes from first tests (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-31 15:53:53 +01:00
Markus Heiser 6274a54746 utils/searx.sh & filtron.sh: misc changes from first tests (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-30 19:55:51 +01:00
Markus Heiser 924bf65517 utils/searx.sh & filtron.sh: misc changes from first tests (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-29 20:00:50 +01:00
Markus Heiser 971a8264b2 utils/searx.sh: add apache site searx.conf:uwsgi (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-27 19:08:40 +01:00
Markus Heiser d171fcd56e utils/searx.sh: add apache site searx.conf:uwsgi 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-21 18:38:57 +01:00
Markus Heiser af2f58fc58 utils/filtron.sh: add script to install filtron middleware (apache) 
Preparation for the installation of an apache site.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-20 19:08:56 +01:00
Markus Heiser 9b5a7f7559 utils/searx.sh: add script to install isolated searx service 
First version which serves searx over uwsgi at http://127.0.0.1:8888

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-20 16:55:05 +01:00
Markus Heiser 89df9d9141 utils/searx.sh: add script to install isolated searx service (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-16 14:01:38 +01:00
Markus Heiser 3cf31528f3 utils/searx.sh: add script to install isolated searx service (WIP) 
WIP: written from scratch / linted but untested

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-14 19:26:54 +01:00
Markus Heiser c2b27d9120 Merge branch 'filtron' of https://github.com/return42/searx into filtron 2020-01-13 18:39:46 +01:00
Markus Heiser b5449ec47c filtron: log suspiciously frequent queries (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-13 18:37:05 +01:00
Markus Heiser 3ed82a97af utils/filtron.sh: add command 'update filtron' 
BTW: tidy up some code style and fix some issues reported by the linter.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-12 15:51:04 +01:00
Markus Heiser 39feb141bc docs(admin): add description of the utils/filtron.sh script 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-11 12:50:40 +01:00
Markus Heiser 4990b07b4b utils/filtron.sh: various fix from first installation test (WIP) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-09 16:25:05 +01:00
Markus Heiser 4139c63d23 utils/filtron.sh: add script to install filtron middleware 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-08 19:21:07 +01:00
Markus Heiser 28dacee288 utils: add lib.sh containing common shell script 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-01-08 18:13:22 +01:00
Markus Heiser d3e4e81faf makefile.sphinx: fix gh-pages / pull before add commits 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-12-22 23:28:07 +01:00
Markus Heiser 121f696460 Makefile: gh-pages: keep history of gh-branch intact (don't drop) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-12-13 12:58:46 +01:00
Markus Heiser ef85943e06 Makefile: add documentation build targets docs & docs-live 
BTW:
- add build & dist folder to .gitignore
- justify indentation of build messages (makefile.python)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-12-12 12:10:32 +01:00
Markus Heiser bb2b9b8da4 utils: add makefile.sphinx to the boilerplate 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-12-12 11:48:51 +01:00
Markus Heiser b55a8004ed boilerplate: add inital Makefile with run & install targets 
Add *Makefile* boilerplate useful for python projects.  All python tasks are
using a virtualenv from ./local/py3

$ make help
  run       - run developer instance
  install   - developer install (./local)
  uninstall - uninstall (./local)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-11-28 19:54:57 +01:00
Dalf c7220b40ea Suggestions: use RawTextQuery to make the suggestions URLs. Update all themes accordingly. 2019-07-17 09:05:29 +02:00
Ivan Skytte Jørgensen 0b7f0bde62 Use a more human-friendly format in searx/data/engines_languages.json (#1399) 
Instead of a single line with 500000 characters use nicely formatted JSON.
Sort the lists in engine_languages.py so when updating it is possible to
more easily see the differences (search engines do change the order their
languages are listed in)
 2018-11-29 16:31:05 +01:00
Alex 50c836864a fetch_firefox_version.py : compatible with Python 3 and minor fixes. 2018-08-05 10:55:42 +02:00
Alexandre Flament 066bd916bf [mod] fetch firefox versions in a standalone script 2018-08-05 10:10:15 +02:00
Marc Abonce Seguin 772c048d01 refactor engine's search language handling 
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.

Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
 2018-03-27 00:08:03 -06:00
Marc Abonce Seguin d1eae9359f fix fetch_langauges to be more accurate 
Add languages supported by either all default general engines or 10 engines.
 2018-03-20 17:58:20 -06:00
cclauss 9984c99665
print() is a function in Python 3 2017-11-14 06:48:54 +01:00
marc 44085e31d0 update engines_languages.json and languages.py 
Also, fix fetch_languages.py so it can run on python3.
 2017-10-10 16:53:28 -05:00
Thirnearez 076cfe25d7 handle input carefully in shell scripts 
- prevent whitespace-splitting of variable expansions
- prevent interpretation of values as flags/options

(mostly)
 2017-10-06 21:00:22 +02:00
NK cda9f8593f Avoid UnicodeEncodeError when redirecting standalone output 2017-07-25 13:39:11 +01:00
marc 805fb02ed1 add language support for qwant 
closes issue #863
 2017-03-18 23:44:21 +01:00
Alexandre Flament 369ec017b7 [enh] standalone_seax.py is a command line interface to searx with JSON output. 2017-01-04 14:14:01 +01:00
Alexandre Flament 7bcdc27a2e [fix] fix standalone_search.py 2017-01-03 19:15:55 +01:00
marc 1175b3906f change language list to only include languages with a minimum of engines 
that support them.
users can still query lesser supported through the :lang_code bang.
 2016-12-29 01:55:30 -06:00
marc 4a1ff56389 minor fixes in utils/fetch_languages.py 2016-12-16 22:14:14 -06:00
marc af35eee10b tests for _fetch_supported_languages in engines 
and refactor method to make it testable without making requests
 2016-12-15 00:40:21 -06:00
marc f62ce21f50 [mod] fetch supported languages for several engines 
utils/fetch_languages.py gets languages supported by each engine and
generates engines_languages.json with each engine's supported language.
 2016-12-13 19:58:10 -06:00
marc 92c6e88ad3 small fixes 2016-12-13 19:56:59 -06:00
marc 922d51bf0c Add country names in language list. 2016-12-13 19:32:43 -06:00
marc a11948c71b Add language support for more engines. 2016-12-13 19:32:43 -06:00
marc c677aee58a filter langauges 2016-12-13 19:32:00 -06:00
marc 149802c569 [enh] add supported_languages on engines and auto-generate languages.py 2016-12-13 19:32:00 -06:00
stepshal 7e16c9d360 Add missing blank lines after class or function definition. 2016-08-11 17:18:12 +07:00
stepshal 6e5f46db42 utils/fetch_currencies: fix comment 2016-07-11 19:54:50 +07:00
stepshal 4261a41c1b Comparison to None should be 'if cond is not None:' 2016-07-11 19:17:05 +07:00
stepshal d9231d173a Fix whitespaces 2016-07-11 18:52:37 +07:00
stepshal cd9b494cb5 Fix quantity of blank lines after code object. 2016-07-10 21:44:27 +07:00
stepshal 2f83750619 Remove trailing whitespace 2016-07-08 23:43:28 +07:00
Alexandre Flament 5525625dae [enh] currency_convert engine: user can write "1 dollars in euros" instead of "1 USD in EUR". 
The currency names are fetched wikidata and store into a static file : searx/data/currencies.json
This file is loaded when the currency_converter is loaded.
A database is perhaps more appropiated.
 2015-05-12 20:52:08 +02:00
Adam Tauber e43f61069a [fix] category name i18n - closes #221 2015-02-10 23:14:37 +01:00
Adam Tauber 9492b50070 [enh] fabfile to setup searx with uwsgi+nginx under debian based distros 2015-02-08 10:42:30 +01:00
Cqoicebordel 5e26ce63be Don't create fuzzy translation 2015-01-05 20:55:56 +01:00
Adam Tauber ea414c6574 [enh] standalone search example 2014-07-21 18:07:04 +02:00
asciimoo 474f6a861d [enh] utils - translation updater script added 2014-02-02 22:23:57 +01:00