Implementations of the *traits* of the engines.
Engine's traits are fetched from the origin engine and stored in a JSON file in
the *data folder*. Most often traits are languages and region codes and their
mapping from SearXNG's representation to the representation in the origin search
engine.
To load traits from the persistence::
searx.enginelib.traits.EngineTraitsMap.from_data()
For new traits new properties can be added to the class::
searx.enginelib.traits.EngineTraits
.. hint::
Implementation is downward compatible to the deprecated *supported_languages
method* from the vintage implementation.
The vintage code is tagged as *deprecated* an can be removed when all engines
has been ported to the *traits method*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch fixes a leftover from [#1548], the list of the plugins was not
up-to-date:
- HTTPS_rewrite has been removed (247c46c6b)
- DOAI_rewrite is named 'Open_Access_DOI_rewrite' (575159b)
[#1548] https://github.com/searxng/searxng/pull/1548
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
With this patch ``searxng.msg`` files can be added to SearXNG. In
``searxng.msg`` files messages can be defined which are not captured by babel's
gettext, like the generic names of the categories or messages that are stored in
constants.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Stuff in folder searxng_extra/ is not suitable for normal users and should only
be used by developers.
The script searxng_extra/standalone_searx.py must not give the impression that
it improves privacy. [1]
[1] https://github.com/searxng/searxng/pull/651#issuecomment-1001389726
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Environment variable to get live builds while modifying CSS & JS of a theme::
LIVE_THEME=simple make run
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Functions implemented to run *Weblate* workflows should use prefix 'weblate.':
- babel.setup.translations.worktree --> weblate.translations.worktree
Create git worktree ${TRANSLATIONS_WORKTREE} and checkout branch
'translations' from Weblate's counterpart (weblate) of the SearXNG
(origin)::
remote weblate https://weblate.bubu1.eu/git/searxng/searxng/
- babel.weblate.to.translations --> weblate.to.translations
Update 'translations' branch of SearXNG (origin) with last additions from
Weblate.
- babel.translations.to.master --> weblate.translations.commit
Update 'translations' branch of SearXNG (origin) with last additions from
Weblate. Copy the changes to the master branch, compile translations and
create a commit in the local branch (master)
- babel.master.to.translations --> weblate.push.translations
Push *translation changes* from SearXNG (origin) to Weblate's
counterpart (weblate).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit 4b43775c9 the brand.git_url & brand.git_branch has been removed, with
this patch these settings are removed from the documentation.
docs/admin/engines/settings.rst
- Remove brand.git_url & brand.git_branch
docs/dev/makefile.rst
- Remove brand.git_url & brand.git_branch
modified docs/dev/lxcdev.rst
- Remove brand.git_url & brand.git_branch
- fix searXNG to SearXNG
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.
It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts. This patch reduce the variables in ./utils/brand.env to what is
needed. The following ('brand:' settings) can be removed from this file:
- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL
Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:
- GIT_URL <--> brand.git_url
- GIT_BRANCH <--> brand.git_branch
- SEARX_URL <--> server.base_url (aka PUBLIC_URL)
- SEARX_PORT <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
modified docs/admin/engines/settings.rst
- Fix documentation and add section 'brand'.
- Add remarks about **buildenv** variables.
- Add remarks about settings from environment variables $SEARX_DEBUG,
$SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET
modified docs/admin/installation-searx.rst & docs/build-templates/searx.rst
Fix template location /templates/etc/searx/settings.yml
modified docs/dev/makefile.rst
Add description of the 'make buildenv' target and describe
- we have all SearXNG setups are centralized in the settings.yml file
- why some tasks need a utils/brand.env (aka instance's buildenv)
modified manage
Settings file from repository's working tree are used by default and
ask user if a /etc/searx/settings.yml file exists.
modified searx/settings.yml
Add comments about when it is needed to run 'make buildenv'
modified searx/settings_defaults.py
Default for server:port is taken from enviroment variable SEARX_PORT.
modified utils/build_env.py
- Some defaults in the settings.yml are taken from the environment,
e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA). When the
'brand.env' file is created these enviroment variables should be
unset first.
- The CONTACT_URL enviroment is not needed in the utils/brand.env
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Docker is blocking network of existing LXC containers / there is a conflict in
the iptables setup of Docker & LXC. With this patch:
- utils/lxc.sh checks internet connectivity (instead of silently hang)
- Chapter "Internet Connectivity & Docker" describes the problem and made a
suggestion for a solution a solution
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Move article 'Developing in Linux Containers' from blog section do developer
section. Since there are no more articles in the blog section, remove the
section completely.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch revision of the article "Engine Overview":
- add links & anchors
- improve formating of the tables
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- Split chapter "Engines" and rename it into "Engines & Settings"
- Move docs/admin/engines.rst -> docs/admin/engines/engine_settings.rst
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch is a a complete revision of the article "Offline engines", which also
merges the content from the searx-wiki [1] into this article.
[1] https://github.com/searx/searx/wiki/Offline-engines
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The article "Offline engines" should be in developer's documentation next to
chapter "Engine overview".
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- pylint searx/engines/xpath.py
- fix indentation of some long lines
- add logging
- add doc-strings
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* searx understand "!ddg !g time" as : send "!g time" to DDG
* !g a DDG bang for Google: DDG return a HTTP redirect to Google
This commit adds a the allows_redirect param not to follow HTTP redirect.
The DDG engine returns a empty result as before without HTTP redirect.
The language_support variable is set to True by default,
and set to False in only 5 engines.
Except the documentation and the /config URL, this variable is not used.
This commit remove the variable definition in the engines, and
set value according to supported_languages length: False when the length is 0,
True otherwise.
Close#2485
Before this commit, in the documentation, the jinja context is 'webapp' and contains
the global variable in the searx.webapp module.
This commit changes this to include only the mandatory variables to build the
documentation.
see searx.search.processors.abstract.EngineProcessor
First the method searx call the get_params method.
If the return value is not None, then the searx call the method search.
check HTTP response:
* detect some comme CAPTCHA challenge (no solving). In this case the engine is suspended for long a time.
* otherwise raise HTTPError as before
the check is done in poolrequests.py (was before in search.py).
update qwant, wikipedia, wikidata to use raise_for_httperror instead of raise_for_status
Add a new parameter "raise_for_status", set by default to True.
When True, any HTTP status code >= 300 raise an exception ( #2332 )
When False, the engine can manage the HTTP status code by itself.
recoll is a local search engine based on Xapian:
http://www.lesbonscomptes.com/recoll/
By itself recoll does not offer web or API access,
this can be achieved using recoll-webui:
https://framagit.org/medoc92/recollwebui.git
This engine uses a custom 'files' result template
set `base_url` to the location where recoll-webui can be reached
set `dl_prefix` to a location where the file hierarchy as indexed by recoll can be reached
set `search_dir` to the part of the indexed file hierarchy to be searched, use an empty string to search the entire search domain
* Made first attempt at the bangs redirects plugin.
* It redirects. But in a messy way via javascript.
* First version with custom plugin
* Added a help page and a operator to see all the bangs available.
* Changed to .format because of support
* Changed to .format because of support
* Removed : in params
* Fixed path to json file and changed bang operator
* Changed bang operator back to &
* Made first attempt at the bangs redirects plugin.
* It redirects. But in a messy way via javascript.
* First version with custom plugin
* Added a help page and a operator to see all the bangs available.
* Changed to .format because of support
* Changed to .format because of support
* Removed : in params
* Fixed path to json file and changed bang operator
* Changed bang operator back to &
* Refactored getting search query. Also changed bang operator to ! and is now working.
* Removed prints
* Removed temporary bangs_redirect.js file. Updated plugin documentation
* Added unit test for the bangs plugin
* Fixed a unit test and added 2 more for bangs plugin
* Changed back to default settings.yml
* Added myself to AUTHORS.rst
* Refacored working of custom plugin.
* Refactored _get_bangs_data from list to dict to improve search speed.
* Decoupled bangs plugin from webserver with redirect_url
* Refactored bangs unit tests
* Fixed unit test bangs. Removed dubbel parsing in bangs.py
* Removed a dumb print statement
* Refactored bangs plugin to core engine.
* Removed bangs plugin.
* Refactored external bangs unit tests from plugin to core.
* Removed custom_results/bangs documentation from plugins.rst
* Added newline in settings.yml so the PR stays clean.
* Changed searx/plugins/__init__.py back to the old file
* Removed newline search.py
* Refactored get_external_bang_operator from utils to external_bang.py
* Removed unnecessary import form test_plugins.py
* Removed _parseExternalBang and _isExternalBang from query.py
* Removed get_external_bang_operator since it was not necessary
* Simplified external_bang.py
* Simplified external_bang.py
* Moved external_bangs unit tests to test_webapp.py. Fixed return in search with external_bang
* Refactored query parsing to unicode to support python2
* Refactored query parsing to unicode to support python2
* Refactored bangs plugin to core engine.
* Refactored search parameter to search_query in external_bang.py
A new option is added to engines to hide error messages from users. It
is called `display_error_messages` and by default it is set to `True`.
If it is set to `False` error messages do not show up on the UI.
Keep in mind that engines are still suspended if needed regardless of
this setting.
Closes#1828
- no more need for a .config.mk
- docs: use searx.brands environment
- searx.sh, filtron.sh & morty.sh are sourcing utils/brand.env
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
We have some variables in the build environment which are also needed in the
grunt process when building themes. Theses variables are relavant if one
creates a fork with its own branding. We treat these variables under the term
'brands'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser
Alexandre Flament
Martin Fischer
Archie
Noémi Ványi
Pierre Chevalier
Adam Tauber
Lukas van den Berk
Noémi Ványi