Commit Graph

3184 Commits

Author SHA1 Message Date
Markus Heiser 809bf1a105 [mod] pylint & document searx.locales (settings.yml: remove locales)
- Add ``# lint: pylint`` header to pylint this python file.
- Fix issues reported by pylint.
- Add source code documentation of modul searx.locales

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-04 07:54:53 +02:00
Alexandre Flament f30d01ffab [mod] settings.yml: remove locales
There are detected from the searx/translations directory
2021-08-03 15:44:45 +02:00
Alexandre Flament bd17544f82 [fix] make test.robot: firefox profile: intl.accept_languages=en
make sure Firefox in test.robot asks for English pages
2021-08-03 15:11:51 +02:00
Alexandre Flament 7fa7177759 Update translations (pot, po) 2021-08-02 09:44:13 +02:00
Markus Heiser e0ca749ff5
Merge pull request #242 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2021-08-02 07:40:19 +00:00
Alexandre Flament b6711ba0ad
Merge pull request #241 from searxng/update_data_update_languages.py
Update searx.data - update_languages.py
2021-08-02 09:01:57 +02:00
Markus Heiser 274edd523c [fix] get common used languages back (min_engines_per_lang = 13)
In commit 4d3f2f48d common used languages has been droped.  By reducing the
number of `min_engines_per_lang` from 15 to 13 we get theses languages back.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-01 13:04:24 +02:00
Markus Heiser 0fc1dcbfa5
Merge pull request #243 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-08-01 09:42:01 +00:00
dalf 7fae5647d3 Update searx.data - update_wikidata_units.py 2021-08-01 06:24:23 +00:00
dalf 2963c83daa Update searx.data - update_currencies.py 2021-08-01 06:24:13 +00:00
dalf 4d3f2f48d2 Update searx.data - update_languages.py 2021-08-01 06:24:12 +00:00
dalf f5389fd722 Update searx.data - update_firefox_version.py 2021-08-01 06:24:01 +00:00
Alexandre Flament 38ee88b390
Merge pull request #229 from dalf/version-from-git
version based on git
2021-07-30 18:57:00 +02:00
Alexandre Flament 4b43775c91 version based on the git repository
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.

It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
2021-07-30 14:40:09 +02:00
Alexandre Flament feef577f99
Merge pull request #230 from dalf/remove-zh-hant-tw
Remove zh_Hant_TW translation
2021-07-28 08:34:09 +02:00
Alexandre Flament 0b2dca088f Remove zh_Hant_TW translation
See https://github.com/searxng/searxng/issues/195#issuecomment-887518808
2021-07-27 18:54:25 +02:00
Alexandre Flament ed0db4d61a
Merge pull request #226 from return42/hardening
[mod] hardening SearXNG instances by default (formats)
2021-07-27 11:22:18 +02:00
Alexandre Flament 24fcd7475a Revert "[brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42"
This reverts commit e946c3462c.
2021-07-27 08:40:53 +02:00
Markus Heiser beff764d74
Merge pull request #216 from return42/fix-searx.sh
normalize environment of installation tasks (shell) with YAML settings
2021-07-27 05:32:57 +00:00
Markus Heiser 80f5241387
Merge pull request #225 from return42/docs-nosql
[docs] document NoSLQ engines MongoDB and redis
2021-07-27 05:32:26 +00:00
Samuel Dudik 7a7ef9cea6 [fix] Seznam engine - some XPath selectors has been changed
Merged from https://github.com/dudik/searx/commit/5a4207759

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-27 07:13:41 +02:00
Markus Heiser 7b40c736e6 [mod] hardening SearXNG instances by default (formats)
Deny formats has been implemented in 6ed4616d.

To harden SearXNG instances by default, other formats than HTML should be
denied.  Most of JSON, RSS and CSV requests are bots [1]::

    Bots are the only users of this feature on a public instance, and they abuse
    it too much that the engines rate limit pretty quickly the IP address of the
    instance.

[1] https://github.com/searxng/searxng/issues/95

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 12:59:09 +02:00
Noémi Ványi 5e648e77ab [docs] document NoSLQ engines MongoDB and redis
Slightly modified merge of commit [3b192e638]

[3b192e638] https://github.com/searx/searx/commit/3b192e638

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-25 11:39:11 +02:00
Alexandre Flament 48fe83b901
Merge pull request #221 from dalf/fix-peertube_fetch_supported_languages
[fix] peertube: update _fetch_supported_languages
2021-07-25 10:30:53 +02:00
Markus Heiser e946c3462c [brand] !!DO NOT MERGE!! this patch uses branch fix-searx.sh @return42
Do not merge this patch in master branch of SearXNG!  This branch exists only
for testing the feature branch fix-searx.sh @return42.

This patch changes the buildenv to::

    GIT_URL='https://github.com/return42/searxng'
    GIT_BRANCH='fix-searx.sh'
    SEARX_PORT='7777'
    SEARX_BIND_ADDRESS='127.0.0.12'

To test installation procedure, clone feature branch (fix-searx.sh)::

    $ cd ~/Downloads
    $ git clone --branch fix-searx.sh https://github.com/return42/searxng searxng
    $ cd searxng

    $ ./utils/searx.sh install all
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser b6a55e223c [mod] reduce enviroment variables in shell scripts to what is needed
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts.  This patch reduce the variables in ./utils/brand.env to what is
needed.  The following ('brand:' settings) can be removed from this file:

- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL

Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:

- GIT_URL            <--> brand.git_url
- GIT_BRANCH         <--> brand.git_branch
- SEARX_URL          <--> server.base_url  (aka PUBLIC_URL)
- SEARX_PORT         <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser fe67f1478f [fix] qwant engine - prevent API locale exception on lang 'all'
Has been reported in [1], error message::

    Error
        Error: searx.exceptions.SearxEngineAPIException
        Percentage: 0
        Parameters: ('API error::locale must be a string,locale must be one of
        the following values: en_gb, en_ie, en_us, en_ca, en_in, en_my, en_au,
        en_nz, cy_gb, gd_gb, de_de, de_ch, de_at, fr_fr, br_fr, fr_be, fr_ch,
        fr_ca, fr_ad, fc_ca, ec_ca, co_fr, es_es, es_ar, es_cl, es_co, es_mx,
        es_pe, es_ad, ca_es, ca_ad, ca_fr, eu_es, eu_fr, it_it, it_ch, pt_br,
        pt_pt, pt_ad, nl_be, nl_nl, pl_pl, zh_hk, zh_cn, fi_fi, bg_bg, et_ee,
        hu_hu, da_dk, nb_no, sv_se, ko_kr, th_th, cs_cz, ro_ro, el_gr',)
        File name: searx/engines/qwant.py:114
        Function: response
        Code: raise SearxEngineAPIException('API error::' + msg)

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 14:48:27 +02:00
Markus Heiser ca57c7421b [fix] qwant engine - prevent exception on date/time value is None
Has been reported in [1], error messages::

  Error
       Error: ValueError
       Percentage: 0
       Parameters: ()
       File name: searx/engines/qwant.py:159
       Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'], None)

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('an integer is required (got type NoneType)',)
        File name: searx/engines/qwant.py:196
        Function: response
       Code: pub_date = datetime.fromtimestamp(item['date'])

Fix timedelta from seconds to milliseconds [1], error message::

    Error
        Error: TypeError
        Percentage: 0
        Parameters: ('unsupported type for timedelta seconds component: NoneType',)
        File name: searx/engines/qwant.py:195
        Function: response
        Code: length = timedelta(seconds=item['duration'])

[1] https://github.com/searxng/searxng/issues/222

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 14:48:14 +02:00
Alexandre Flament b0a12924a0 [fix] peertube: update _fetch_supported_languages
update the regex to match the changes in peertube source code
fix "make data.languages"
2021-07-23 12:03:16 +02:00
Alexandre Flament b1100e2b7d [fix] /preferences: remove the empty engine category "social medias" (it is "social media") 2021-07-23 10:43:16 +02:00
Markus Heiser 2964a24b3d [fix] utils/build_env.py and add documentation
modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:54 +02:00
Markus Heiser 6fbf5180bf [mod] drop obsolete searx.brand namespace
The usages of the searx.brand namespace has been removed, the searx.brand
namespace is now longer needed.

The searx.brand namespace was an interim solution which has been added in commit
9e53470b4, see commit message there ...

    This patch is a first 'proof of concept'.  Later we can decide to remove the
    brand namespace entirely or not.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser 3f638ed196 [mod] drop usage of the searx.brand namespace (templates & /config)
In the templates and the /config (JSON) the usage of the 'brand.*' name
space is replaced by 'searx.get_setting' function.

- new_issue_url          --> get_setting('brand.new_issue_url')
- brand.GIT_URL          --> get_setting('brand.git_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.DOCS_URL         --> get_setting('brand.docs_url')
- brand.ISSUE_URL        --> get_setting('brand.issue_url')
- brand.CONTACT_URL      --> get_setting('general.contact_url', '')

The macro 'new_issue' from searx/templates/*/messages/no_results.html
is now imported with context::

    {% from '__common__/new_issue.html' import new_issue with context %}

To get *public instances URL* from context's 'get_setting()' function::

    get_setting('brand.public_instances','')

Macro's prototype does no longer need the 'new_issue_url' argument and has been
changed to::

    macro new_issue(engine_name, engine_reliability)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser c9220de690 [mod] drop unused setting option brand:twitter_url
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser 2a2b6c4d57 [fix] add missing brand options to the settings SCHEMA
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser 3e50e8de3e [mod] drop usage of the searx.brand namespace (python procs)
Added function searx.get_setting(name, default=_unset):
  Returns the value to which ``name`` point.  If there is no such name in the
  settings and the ``default`` is unset, a KeyError exception is raised.

In all the python processes ..

- make docs
- make buildenv
- make install (setup.py)

the usage of the 'brand.*' name space is replaced by 'searx.get_setting'
function.

- brand.SEARX_URL        --> get_setting('server.base_url')
- brand.GIT_URL          --> get_setting('brand.git_url')
- brand.GIT_BRANCH'      --> get_setting('server.base_url')
- brand.ISSUE_URL        --> get_setting('brand.issue_url')
- brand.DOCS_URL         --> get_setting('brand.docs_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.CONTACT_URL      --> get_setting('general.contact_url', '')
- brand.WIKI_URL         --> get_setting('brand.wiki_url')
- brand.TWITTER_URL      --> get_setting('brand.twitter_url', '')

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-18 15:55:42 +02:00
Alexandre Flament f523fd3ea7
Merge pull request #211 from MarcAbonce/onions_v3_fix_searxng
Update onion engines to v3
2021-07-16 17:25:37 +02:00
Alexandre Flament d47b8e36cf
Merge pull request #207 from return42/mongodb
[enh] add mongodb offline engine
2021-07-16 16:15:01 +02:00
Alexandre Flament 0d65a81b1c [mod] qwant engine: fix typos / minor change
minor modification of commit 628b5703f3
(no functionnal change)
2021-07-16 15:32:12 +02:00
Marc Abonce Seguin 1b05ea6a6b update onion engines to v3
remove not_evil which has been down for a while now:
https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
2021-07-16 01:36:34 -07:00
Markus Heiser 0a9cd08bf1 [enh] add mongodb offline engine
Cherry-Pick: https://github.com/searx/searx/commit/198aad43
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-15 21:35:33 +02:00
Markus Heiser 628b5703f3 [mod] improve video results of the qwant engine
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-15 20:10:37 +02:00
Alexandre Flament f376b4ed3e
Merge pull request #205 from unixfox/patch-2
Add missing parameter for mobile UI search
2021-07-15 17:19:12 +02:00
Émilien Devos 6c9f276571
Add missing parameter for mobile UI search 2021-07-15 13:00:32 +00:00
Markus Heiser 420e3caba3 [mod] enable Qwant engines (web, news, images, video) by default
Qwant is a fast and reliable search engine and AFAIK there is no CAPTCHA.  Let
us enable Qwant engines by default.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14 09:47:32 +02:00
Markus Heiser ef6e1bd6b9 [fix] Qwant engines - implement API v3 and add 'quant videos'
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.

This implementation is used by different qwant engines in the settings.yml::

  - name: qwant
    categories: general
    ...
  - name: qwant news
    categories: news
    ...
  - name: qwant images
    categories: images
    ...
  - name: qwant videos
    categories: videos
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14 09:47:32 +02:00
Markus Heiser 2767046065 [fix] drop engine dogpile
The engine was added in commit a4b07460 but now it shows new issues [1].

In the 90'th of the last century, dogpile had its own WEB index, but nowadays it
is a meta-search engine [2]

  Powered by technology, Dogpile returns all the best results from leading
  search engines including Google and Yahoo!

Using dogpile as an engine in SearXNG needs more investigation, a XPath solution
like we have is not enough.   It is questionable whether it still makes sense to
investigate more into a meta-search engine with a ReCAPTCHA in front.

With this patch the dogpile engine is removed

[1] https://github.com/searxng/searxng/issues/202
[2] https://www.dogpile.com/support/aboutus

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-13 14:42:09 +02:00
Markus Heiser 513c73a309 [drop] engine torrentz: torrentz2.eu and torrentz2.is are offline
[1] https://torrentfreak.com/torrentz2-eu-domain-suspended-by-registry-on-public-prosecutors-order-200628/

Suggested-by: @rasos https://github.com/searx/searx/issues/1875#issuecomment-877755872
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-11 13:24:33 +02:00
Émilien Devos d9d9bd720d
Fix google images
Proposed fix in https://github.com/searx/searx/pull/2115#issuecomment-876716010
2021-07-10 14:09:29 +00:00
Allen 3b2e0a93b7 [enh] add gpodder.net (JSON)
Engine just for Podcasts.  An API which returns Podcasts and their Info like:
website, author etc.

Upstream query example: https://gpodder.net/search.json?q=linux
2021-07-03 22:30:20 +02:00
Markus Heiser d5eedf1b61 [yamllint] fix some warning issues reported by yamllint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 21:55:36 +02:00
Allen 9648d71ad8 [enh] add woxikon.de synonyme (Xpath)
Added synonyme.woxikon.de using the xpath engine.  Adds a site which returns
word synonyms although just in German.

Depending on the query not all synonyms are shown because of not the best xpath
selection.  But should do the job just fine.

Upstream example query: https://synonyme.woxikon.de/synonyme/test.php
2021-07-03 21:51:26 +02:00
Markus Heiser bb98a07b50
Merge pull request #196 from return42/fix-brave
[fix] brave engine: shows descriptions with their correct URLs
2021-07-03 19:44:48 +00:00
Markus Heiser 9d97b2e854 [fix] brave engine: shows descriptions with their correct URLs
BTW add about section to the YAML configuration

It now shows descriptions with their correct URLs when there are videos in the
search results, pulling content_xpath from snippet-description instead of
snippet-content.

Suggested-by: @eagle-dogtooth https://github.com/searx/searx/issues/2857#issuecomment-869119968
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 20:57:55 +02:00
Markus Heiser 7e8d1f9935
Merge pull request #178 from return42/whaleslide.com
[enh] Add WhaleSlide engine (JSON)
2021-07-03 16:22:47 +00:00
Markus Heiser 24f2376c11 [pylint] prepare for pylint v2.9.3 / fix some (new) pylint issues
Upgrade from pylint v2.8.3 to 2.9.3 raise some new issues::

  searx/search/checker/__main__.py:37:26: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
  searx/search/checker/__main__.py:38:26: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
  searx/search/processors/__init__.py:20:0: R0402: Use 'from searx import engines' instead (consider-using-from-import)
  searx/preferences.py:182:19: C0207: Use data.split('-', maxsplit=1)[0] instead (use-maxsplit-arg)
  searx/preferences.py:506:15: R1733: Unnecessary dictionary index lookup, use 'user_setting' instead (unnecessary-dict-index-lookup)
  searx/webapp.py:436:0: C0206: Consider iterating with .items() (consider-using-dict-items)
  searx/webapp.py:950:4: C0206: Consider iterating with .items() (consider-using-dict-items)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 17:54:08 +02:00
Alexandre Flament 961dd287a1
Merge pull request #185 from searxng/update_data_update_external_bangs.py
Update searx.data - update_external_bangs.py
2021-07-01 18:43:18 +02:00
Alexandre Flament 508195a850
Merge pull request #186 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-07-01 18:41:54 +02:00
Alexandre Flament 89a27f4f96
Merge pull request #187 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-07-01 18:41:43 +02:00
dalf 973d507410 Update searx.data - update_ahmia_blacklist.py 2021-07-01 06:22:48 +00:00
dalf f6a8a5d017 Update searx.data - update_firefox_version.py 2021-07-01 06:22:48 +00:00
dalf 77a432cc95 Update searx.data - update_wikidata_units.py 2021-07-01 06:22:43 +00:00
dalf 726ad776a3 Update searx.data - update_external_bangs.py 2021-07-01 06:22:24 +00:00
Alexandre Flament 30e4a2a224
Merge pull request #174 from searxng/static_script
[mod] add make targets to manage the build files in the /searx/static directory
2021-06-28 10:53:32 +02:00
Alexandre Flament d29094579b [build] /static 2021-06-28 10:39:10 +02:00
Markus Heiser 76a066d4a3 [mod] themes: remove banner generated by grunt
The banner is useless and responsible for unwanted diff.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-28 10:38:44 +02:00
Alexandre Flament 7308709b8b
Merge pull request #179 from return42/brave
[enh] add Brave-Search engine (XPath)
2021-06-28 09:23:25 +02:00
Markus Heiser 6787e5a36b [fix] decoding of saved preferences in the URL
To compress saved preferences in the URL was introduced in 5f758b2d3 and
slightly fixed in 8f4401462.  But the main fail was not fixed; The decompress
function returns a binary string and this binary should first be decoded to a
string before it is passed to urllib.parse_qs.

BTW: revert the hot-fix from 5973491

Related-to: https://github.com/searxng/searxng/issues/166
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 17:14:13 +02:00
Markus Heiser e0365d987b [enh] add Brave-Search engine (XPath)
Add https://search.brave.com which was requested by @kaonashi696 [1].
This patch was suggested by @eagle-dogtooth [2].

[1] https://github.com/searx/searx/issues/2857
[2] https://github.com/searx/searx/issues/2857#issuecomment-867199241

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 15:54:45 +02:00
Allen e2094cb226 [enh] Add WhaleSlide engine (JSON)
Add WhaleSlide engine [1] / merged from [2]

[1] https://whaleslide.com
[2] https://github.com/searx/searx/pull/2861
2021-06-26 15:28:44 +02:00
Émilien Devos 63ebe00554
Set use_mobile_ui to false for Google engine 2021-06-26 09:58:35 +00:00
Markus Heiser a3f58a3a3d [emacs] .dir-locals.el: add setup for js-mode
- move jshint option from gruntfile to .jshintrc
- remove trailing-whitespace from gruntfile and
- add jshint esversion: 6
- .dir-locals.el add locals for js-mode to use JSHint from the simple theme

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Markus Heiser f73a00dcee [stylelint] disable role 'no-descending-specificity'
This patch disables role 'no-descending-specificity'.  IMO it is better to have
this rule active (see below [1]), but it is hard to rewrite the less files to
pass this rule, so for the first I chose to disable this rule.

---

Source order is important in CSS, and when two selectors have the same
specificity, the one that occurs last will take priority. However, the situation
is different when one of the selectors has a higher specificity. In that case,
source order does not matter: the selector with higher specificity will win out
even if it comes first.

The clashes of these two mechanisms for prioritization, source order and
specificity, can cause some confusion when reading stylesheets. If a selector
with higher specificity comes before the selector it overrides, we have to think
harder to understand it, because it violates the source order
expectation. Stylesheets are most legible when overriding selectors always come
after the selectors they override. That way both mechanisms, source order and
specificity, work together nicely.

This rule enforces that practice as best it can, reporting fewer errors than it
should. It cannot catch every actual overriding selector, but it can catch
certain common mistakes.

[1] https://stylelint.io/user-guide/rules/list/no-descending-specificity/

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Markus Heiser 0b7d03c694 [stylelint] simple theme: fix some errors reported by stylelint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Markus Heiser 14b09c15c4 [fix] simple theme: use stylint to fix common lint errors
This fix was autogenerated by::

     npx stylelint -f unix --fix 'searx/static/themes/simple/src/less/**/*.less'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Markus Heiser dfc5c3bc15 [mod] add stylelint process to target themes.simple
Before build, do lint the LESS files using stylelint [1].  The configuration
'stylelint-config-standard' [2] is added to the root of the simple theme [3].

[1] https://stylelint.io/
[2] https://github.com/stylelint/stylelint-config-standard
[3] https://stylelint.io/user-guide/configure

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-24 13:27:21 +02:00
Alexandre Flament 2530576f24
Merge pull request #157 from searxng/mod_static_build
static build: move generated files to searx/static/themes/*/src/generated
2021-06-22 10:36:19 +02:00
Alexandre Flament 04492b76de [themes] ignore packages-lock.json 2021-06-22 08:17:06 +02:00
Alexandre Flament 2bc9b00cd7 Static build 2021-06-22 08:15:17 +02:00
Alexandre Flament dca3bcca9e [mod] simple theme: include fonts
"npm run webfont" to build the fonts directory.
It requires fontforge and ttfautohint distro packages.

partial revert of commit 7137d2893f
2021-06-22 08:15:17 +02:00
Alexandre Flament d20f6a1f19 [mod] make themes.all update pygments*.less
Add a searx/static/themes/*/src/generated empty folder
2021-06-22 08:15:11 +02:00
Markus Heiser 5973491701 [fix] UnicodeEncodeError for preferences ULR
Closes: https://github.com/searxng/searxng/issues/166

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22 01:35:54 +02:00
Markus Heiser 0ef6aa5126 [docs] add documentation from the sources of the google engines
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 18:25:52 +02:00
Markus Heiser 05e90f2e57 [fix] google answers: normalize space of the answers.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 16:50:25 +02:00
Markus Heiser f096d68ec6 [mod] google engine: reduce mobile UI parameters to what is needed
Reverse engineering shows that not all of the parameters used by google's mobile
UI (aka "more results" button) are needed [1].

[1] https://github.com/searxng/searxng/pull/160#issuecomment-865013625

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-21 16:50:23 +02:00
Alexandre Flament 7a5c36408a [mod] google: add "use_mobile_ui" parameter to use mobile endpoint.
disable by default, it has to be enabled in settings.yml

related to  #159
2021-06-21 14:52:04 +02:00
Alexandre Flament bb3e67f72b [mod] simple theme: normalize.css becomes a packages.json dependency 2021-06-21 08:28:29 +02:00
Markus Heiser 9328c66e93 [fix] google news - send CONSENT Cookie to not be redirected
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking.  To get the consent
from the user, google-news requests are redirected to confirm and get a CONSENT
Cookie from https://consent.google.de/s?continue=...

This patch adds a CONSENT Cookie to the google-news request to avoid
redirection.

The behavior of the CONTENTS cookies over all google engines seems similar but
the pattern is not yet fully clear to me, here are some random samples from my
analysis ..

Using common google search from different domains::

    google.com:        CONSENT=YES+cb.{{date}}-14-p0.de+FX+816
    google.de:         CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    google.fr:         CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826

When searching about videos (google-videos)::

    google.es:         CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076
    google.de:         CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171

Google news has only one domain for all languages::

    news.google.com:   CONSENT=YES+cb.{{date}}-14-p0.de+FX+816

Using google-scholar search from different domains::

    scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
    scholar.google.fr: does not use such a cookie / did not ask the user
    scholar.google.es: does not use such a cookie / did not ask the user

Interim summary:

  Pattern is unclear and I won't apply the CONSENT cookie to all google engines.
  More experience is need before we generalize the CONSENT cookies over all
  google engines.

Related:

- e9a6ab401 [fix] youtube - send CONSENT Cookie to not be redirected
- https://github.com/benbusby/whoogle-search/issues/311
- https://github.com/benbusby/whoogle-search/issues/243

[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-18 13:21:20 +02:00
Markus Heiser dd7b53d369 [fix] google-news engine - KeyError: 'hl in request
Since we added

- 1c67b6aec [enh] google engine: supports "default language"

there is a KeyError: 'hl in request,error pattern::

    ERROR:searx.searx.search.processor.online:engine google news : exception : 'hl'
    Traceback (most recent call last):
      File "searx/search/processors/online.py", line 144, in search
        search_results = self._search_basic(query, params)
      File "searx/search/processors/online.py", line 118, in _search_basic
        self.engine.request(query, params)
      File "searx/engines/google_news.py", line 97, in request
        if lang_info['hl'] == 'en':
      KeyError: 'hl'

Closes: https://github.com/searxng/searxng/issues/154
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-18 11:34:11 +02:00
Alexandre Flament 881659ca9d [mod] oscar theme: /preferences : HTML detail order match visual tabs
First details about the general tab, then detail about UI tab, etc...
No functionnal change
2021-06-17 15:29:07 +02:00
Alexandre Flament f83b64270c [mod] oscar theme /preferences: reorganize the preferences
close #115
2021-06-17 15:29:07 +02:00
Alexandre Flament 92c68fe636
Merge pull request #151 from searxng/mod_oscar
make themes: 2 node_modules directories instead of 3
2021-06-17 10:46:50 +02:00
Markus Heiser 96d4dbf74d [fix] typo in a searx/settings.yml
remove the erroneously leading underline

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-16 16:51:28 +02:00
Alexandre Flament 14ba56fd1a [mod] make node.clean: call the "clean" script from the packages.json
The dependencies and build process are described in the gruntfiles.js and packages.json files.
It makes sense to move the clean up to these files too.
2021-06-16 16:04:58 +02:00
Alexandre Flament c013256881 [mod] remove common grunt install
Before this commit, there are 3 node_modules directory:
* one in .
* two others in ./searx/statics/themes/*

This is no desirable:
* it declares the npm depdenencies in the shell script.
* dependabot can't updates theses dependencies.
* this is a not standard way to build a package (two different locations for the dependencies).

With this commit and the PR  #150 there is one unique node_modules directory per theme.
2021-06-16 15:39:32 +02:00
Alexandre Flament 85033f3d09 Static build 2021-06-16 14:30:46 +02:00
Alexandre Flament 7137d2893f [mod] simple theme: remove src/less/ion.less
This file is generated by webfont.
* It is now generated as searx/static/themes/simple/ion.less
* It is generated before the .less compilation.
* .gitignore includes this file

Add two new package depedencies: fontforge ttfautohint
See utils/searx.sh
2021-06-16 14:30:09 +02:00
Alexandre Flament e3ebfd41d7 [mod] simple theme: autocomplete-js becomes a packages.json dependency
Update to version 2.7.1
2021-06-16 12:44:07 +02:00
Alexandre Flament b4a47cacd8 [mod] simple theme: leaflet becomes a packages.json dependency
the build of the themes updates:
* js/leaflet.js ( was leaflet/leaflet.js )
* css/leaflet.css ( was leaflet/leaflet.css )
* css/images ( was leaflet/images )
2021-06-16 12:44:07 +02:00
Alexandre Flament 6288b549e2 [mod] simple theme: move magnet.svg to src/fonts
see https://github.com/searx/searx/pull/1109
2021-06-16 12:44:07 +02:00
Alexandre Flament 6b80c57a3c [mod] simple theme: move source files to the src directory 2021-06-16 12:38:06 +02:00
Alexandre Flament 49ea5b7644 Static build 2021-06-16 09:16:37 +02:00
Alexandre Flament 9108c90f0e [mod] simple theme: update dependencies 2021-06-16 09:16:02 +02:00
Markus Heiser 343570f7fb [pylint] searx/engines/duckduckgo_definitions.py
BTW: normalize indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-14 09:22:29 +02:00
Markus Heiser 2449ea70b5
Merge pull request #139 from return42/refactor-webapp
[coding-style] searx/webapp.py - normalize indentations
2021-06-13 09:19:27 +00:00
Markus Heiser 2ac3e5b20b [fix] log messages from: google- images, news, scholar, videos
- HTTP header Accept-Language --> lang_info['headers']['Accept-Language']
- remove obsolete query_url log messages which is already logged by
  httpx._client:HTTP request

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-11 16:31:50 +02:00
Markus Heiser 1ac3961336 [mod] google - get_lang_info add documentataion & comments
BTW: remove obsolete log messages from google engine

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-11 16:06:36 +02:00
Alexandre Flament 1c67b6aece [enh] google engine: supports "default language"
Same behaviour behaviour than Whoogle [1].  Only the google engine with the
"Default language" choice "(all)"" is changed by this patch.

When searching for a locate place, the result are in the expect language,
without missing results [2]:

  > When a language is not specified, the language interpretation is left up to
  > Google to decide how the search results should be delivered.

The query parameters are copied from Whoogle.  With the ``all`` language:

- add parameter ``source=lnt``
- don't use parameter ``lr``
- don't add a ``Accept-Language`` HTTP header.

The new signature of function ``get_lang_info()`` is:

    lang_info = get_lang_info(params, lang_list, custom_aliases, supported_any_language)

Argument ``supported_any_language`` is True for google.py and False for the other
google engines.  With this patch the function now returns:

- query parameters: ``lang_info['params']``
- HTTP headers: ``lang_info['headers']``
- and as before this patch:
  - ``lang_info['subdomain']``
  - ``lang_info['country']``
  - ``lang_info['language']``

[1] https://github.com/benbusby/whoogle-search
[2] https://github.com/benbusby/whoogle-search/releases/tag/v0.5.4
2021-06-10 10:22:01 +02:00
Markus Heiser d0daa87ce6 [coding-style] searx/webapp.py - normalize indentations
No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-10 09:35:00 +02:00
Alexandre Flament 73dbee45a6
Merge pull request #90 from searxng/enh-osm-engine
[enh] improve openstreetmap engine
2021-06-10 09:30:41 +02:00
Markus Heiser 15178e5b96 [static] make themes.all - based on commit 12af3b39
Based on commit 12af3b39 this patch is generated by::

    make themes.all

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-10 09:20:51 +02:00
Markus Heiser 12af3b39db [fix] make simple/result_templates/map.html more CSP compliant [1]
[1] https://github.com/searxng/searxng/issues/57

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-09 18:08:23 +02:00
Markus Heiser bf10b4a857 [fix] openstreetmap - fix some minor whitespace & indentation issues
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-09 18:08:23 +02:00
Markus Heiser 5cf1ae2672 [pylint] searx/data/__init__.py
BTW: add doc strings and moved __all__ to the top [1]

[1] https://www.python.org/dev/peps/pep-0008/#module-level-dunder-names

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-09 18:08:23 +02:00
Alexandre Flament 2e5d823162 [fix] searx/data/__init__.py: rename __init__ as __all__ 2021-06-09 18:08:23 +02:00
Alexandre Flament c75425655f [enh] openstreetmap / map template: improve results
implements ideas described in #69

* update the engine
* use wikidata
* update map.html template
2021-06-09 18:08:23 +02:00
Alexandre Flament 92c8a8829f [fix] strip spaces from searx user agent
h11 (used by httpx) rejects HTTP request with a trailing space in HTTP headers
2021-06-09 18:08:23 +02:00
Alexandre Flament 2a109d0a5f [mod] refactor searx.webapp.render
render automatically adds these variables to the template context:
* advanced_search
* all_categories
* categories
	before render was checking if the variable was already set
	but it is actually never set by the callers
2021-06-08 15:54:11 +02:00
Alexandre Flament 70bbd14b1a [mod] move hook to set Unix thread name into searx.unixthreadname
requires setproctitle (but optional)
2021-06-08 15:54:11 +02:00
Alexandre Flament 0e23113f70 [mod] remove unused filter searx.webapp.extract_domain
it was used in the pix-art theme
2021-06-08 15:54:11 +02:00
Alexandre Flament cfa07b0a9a [mod] refactor searx.webapp.render
no functional change, reorder declarations
2021-06-08 15:54:11 +02:00
Alexandre Flament 91856e8f6a [mod] refactor searx.webapp.render: remove unused parameters 2021-06-08 15:54:11 +02:00
Alexandre Flament 15567955b4 [mod] remove check of Python 2. 2021-06-08 15:54:11 +02:00
Alexandre Flament aedf0aa5fe [mod] remove searx.webapp.get_base_url function
see the result of:
git grep "base_url" searx/templates
"base_url" is not used any more.
2021-06-08 15:54:11 +02:00
Alexandre Flament eeb0998787 [mod] refactor: move Flask proxy fix to searx.flaskfix module 2021-06-08 15:54:11 +02:00
Markus Heiser 5c5db719d2
Merge pull request #97 from return42/drop-searx-admin
[docs] reorder blog articles
2021-06-08 10:56:18 +00:00
Markus Heiser 5a7b12ee6f [yamllint] searx/settings.yml
Fixed messages reported by::

    make test.yamllint

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-05 17:41:24 +02:00
Markus Heiser 4e845966ba [yamllint] searx/settings_robot.yml
Fixed messages reported by::

    make test.yamllint

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-05 17:41:24 +02:00
Alexandre Flament cf2b549967 Update engines_languages.json 2021-06-04 16:17:22 +02:00
Alexandre Flament 8194db4e21 [fix] peertube fetch supported languages
close #127
2021-06-04 16:17:20 +02:00
Markus Heiser f122cb0e27 [fix] typo: online_dictionnary --> online_dictionary
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04 15:05:58 +02:00
Markus Heiser 79cc82a4db [docs] add engine "Demo Online Engine"
This engine just exists for documentation purpose.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04 15:05:58 +02:00
Markus Heiser 1c8cf1d3a8 [docs] add engine "Demo Offline Engine"
This engine just exists for documentation purpose.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-04 15:04:38 +02:00
Markus Heiser d910519ba1
Merge pull request #128 from searxng/debug-settings-reload-on-change
[enh] flask debug mode: reload the app when searx/settings.yml changes
2021-06-04 05:34:11 +00:00
Alexandre Flament a7d575c099 [enh] flask debug mode: reload the app when searx/settings.yml changes 2021-06-03 12:03:38 +02:00
Alexandre Flament 7457f3fe40
Merge pull request #124 from return42/searx-merge
merge redis offline engine from searx
2021-06-02 12:35:33 +02:00
Markus Heiser 39c18274c6 [fix] enigine redis - avoid error when the engine is loaded
Should be _redis_client to avoid an error when the engine is loaded.

Suggested-by: @dalf https://github.com/searxng/searxng/pull/124#pullrequestreview-673885664
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-02 09:54:58 +02:00
Markus Heiser 60dc91f234 [static] make /stats more CSP compliant- make themes.all
Based on commit:

- a89b823f [mod] remove overpass API call

this patch is generated by::

     make themes.all

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01 17:54:31 +02:00
Alexandre Flament a89b823f8d [mod] remove overpass API call
prepare the code the PR #90
2021-06-01 17:52:49 +02:00
Alexandre Flament 8375974dff [fix] sys.exit(1) when there is duplicate engine name 2021-06-01 16:37:20 +02:00
Markus Heiser 8908937046 [mod] searx.engines.load_engine return None instead of sys.exit(1)
Loading an engine should not exit the application (*). Instead
of exit, return None.

(*) RuntimeError still exit the application: syntax error, etc...

BTW: add documentation and normalize indentation (no functional change)

Suggested-by: @dalf https://github.com/searxng/searxng/pull/116#issuecomment-851865627
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01 16:35:17 +02:00
Alexandre Flament 70a9208972 [mod] searx.engines.__init__: refactoring 2021-06-01 16:32:40 +02:00
Alexandre Flament e3f4a77311
Merge pull request #110 from searxng/mod-default-settings
[mod] move all default settings into searx.settings_defaults
2021-06-01 16:13:04 +02:00
Markus Heiser 495d86b520 [pylint] searx/__init__.py & searx/settings_defaults.py
BTW: normalize some indentations in searx/settings_defaults.py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01 16:03:19 +02:00
Adam Tauber e4b6558339 [enh] add redis offline engine / https://redis.io/
Slightly modified merge of commit [97269be6], [01a8a5814a] and [c8d2b5eb] from
searx.

[97269be6] https://github.com/searx/searx/commit/97269be6
[01a8a581] https://github.com/searx/searx/commit/01a8a581
[c8d2b5eb] https://github.com/searx/searx/commit/c8d2b5eb

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01 11:51:25 +02:00
Adam Tauber e33858d6ef [fix] correct kv template formatting and remove internal data
Slightly modified merge of commit [ea7ccf24] from searx.

[ea7ccf24] https://github.com/searx/searx/commit/ea7ccf24

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-01 10:53:37 +02:00
Alexandre Flament 925c21ce42
Merge pull request #122 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-06-01 10:03:50 +02:00
Alexandre Flament 3266738e7a
Merge pull request #121 from searxng/update_data_update_currencies.py
Update searx.data - update_currencies.py
2021-06-01 10:03:35 +02:00
Alexandre Flament 54ae95b95b
Merge pull request #120 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-06-01 10:03:22 +02:00
dalf 43d06e8224 Update searx.data - update_ahmia_blacklist.py 2021-06-01 08:01:50 +00:00
dalf e53ed4742c Update searx.data - update_currencies.py 2021-06-01 08:01:48 +00:00
dalf 666a25314a Update searx.data - update_wikidata_units.py 2021-06-01 08:01:45 +00:00
dalf 9f3a455777 Update searx.data - update_firefox_version.py 2021-06-01 08:01:33 +00:00
Alexandre Flament 6fa114c9ba [mod] settings_default: remove searx.search.max_request_timeout global variable 2021-06-01 08:10:15 +02:00
Alexandre Flament 4b07df62e5 [mod] move all default settings into searx.settings_defaults 2021-06-01 08:10:15 +02:00
Alexandre Flament 6ca74386d6
[fix] mankier: fix paging
See https://github.com/searxng/searxng/pull/111#issuecomment-851641967
2021-05-31 22:33:40 +02:00
Kyle Anthony Williams d6a2d4f969 [enh] add engine - Docker Hub
Slightly modified merge of commit [1cb1d3ac] from searx [PR 2543]:

      This adds Docker Hub .. as a search engine .. the engine's favicon was
      downloaded from the Docker Hub website with wget and converted to a PNG
      with ImageMagick .. It supports the parsing of URLs, titles, content,
      published dates, and thumbnails of Docker images.

[1cb1d3ac] https://github.com/searx/searx/pull/2543/commits/1cb1d3ac
[PR 2543] https://github.com/searx/searx/pull/2543

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-30 15:18:36 +02:00
Alexandre Flament 1113f7e616 [mod] the bittorent search engines are available only in the files category
related to #101
2021-05-29 16:14:19 +02:00
Alexandre Flament e239369f1f
Merge pull request #109 from return42/sql-engines
Merge MySQL and PostgreSQL from searx
2021-05-29 08:34:19 +02:00
Alexandre Flament c22c7cf35b
Merge pull request #111 from return42/mankier
[enh] add engine ManKier (JSON) / ManKier: Linux man pages
2021-05-29 08:33:54 +02:00
Jordan Webb a8f474b23f [enh] add engine PyPi (XPath)
Add a XPath configuration for PyPI (Python Package Index)

Slightly modified merge of [66d06b05] from searx's [PR 2830].

[PR 2830] https://github.com/searx/searx/pull/2830
[66d06b05] https://github.com/searx/searx/pull/2830/commits/66d06b05

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-29 05:51:33 +02:00
Jordan Webb 52672d887b [enh] add engine ManKier (JSON) / ManKier: Linux man pages
Slightly modified merge of [a881e742aa] from searx's [PR 2829].

[PR 2829] https://github.com/searx/searx/pull/2829
[a881e742aa] https://github.com/searx/searx/pull/2829/commits/a881e742aa

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-29 05:29:47 +02:00
Noémi Ványi 87a01a1736 [enh] add MySQL engine
Slightly modified merge of [c00a33fe] from searx.

[c00a33fe] c00a33feee

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 17:36:46 +02:00
Noémi Ványi 324aa96062 [enh] add PostgreSQL engine
Slightly modified merge of [22079ff] from searx.

[22079ff] 22079ffdef

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 17:34:44 +02:00
Allen a4b0746044 [enh] add metasearch engin dogpile.com (XPath)
Add Dogpile engine [1] / merged from [2]

[1] https://www.dogpile.com
[2] https://github.com/searx/searx/pull/2822
2021-05-28 15:52:28 +02:00
LL Productions FR 15f1d6a80f [enh] add engine Wikimini (fr.wikimini.org)
Online encyclopedia for children (only language fr), merged from [1]

[1] https://github.com/searx/searx/pull/2819
2021-05-28 14:03:21 +02:00
Alexandre Flament 83ccf7b04d
Merge pull request #99 from return42/webapp-misc
[enh] add settings option to enable/disable search formats
2021-05-28 13:26:48 +02:00
Markus Heiser 96b223023a [mod] utils.get_value() - avoidance of a recursion
In a comment [1] dalf suggested to avoid a recursion of get_value()

[1] https://github.com/searxng/searxng/pull/99#discussion_r640833716

Suggested-by: Alexandre Flament <alex@al-f.net>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 08:32:52 +02:00
Markus Heiser 6ed4616da9 [enh] add settings option to enable/disable search formats
Access to formats can be denied by settings configuration::

    search:
        formats: [html, csv, json, rss]

Closes: https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28 08:32:52 +02:00
Alexandre Flament b48b4c93d5
Merge pull request #100 from return42/webapp-pylint
[pylint] webapp.py
2021-05-27 17:04:32 +02:00
Markus Heiser 32b5a0ef7b
Merge pull request #93 from return42/genius-misc
Some minor Genius improvements
2021-05-27 14:23:22 +00:00
Markus Heiser 25b5797a0c
Merge pull request #103 from searxng/add-sqlite-engine2
[enh] add offline engine for sqlite database
2021-05-27 14:06:42 +00:00
Alexandre Flament b1da97d092
Merge pull request #96 from return42/unsplash
lint & fix unsplash engine
2021-05-27 14:58:59 +02:00
Alexandre Flament 2ea34a3c36 [enh] add offline engine for sqlite database
To test & demonstrate this implementation download:

  https://liste.mediathekview.de/filmliste-v2.db.bz2

and unpack into searx/data/filmliste-v2.db, in your settings.yml define a sqlite
engine named "demo"::

    - name : demo
      engine : sqlite
      shortcut: demo
      categories: general
      result_template: default.html
      database : searx/data/filmliste-v2.db
      query_str :  >-
        SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
               COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
               description AS content
          FROM film
         WHERE title LIKE :wildcard OR description LIKE :wildcard
         ORDER BY duration DESC
      disabled : False

Query to test: "!demo concert"

This is a rewrite of the implementation from commit [1]

[1] searx/searx@8e90a21

Suggested-by: @virtadpt searx/searx#2808
2021-05-27 14:27:11 +02:00
Markus Heiser e1f244b2d5 [pylint] webapp.py - fix all messages from pylint
No functional change, just some linting.

- fix messages from pylint (see below)
- log where general Exceptions are catched (broad-except)
- normalized various indentation
- To avoid clashes with common names, add prefix 'route_' to all @app.route
  decorated functions.

Fixed messages::

    searx/webapp.py:744:0: C0301: Line too long (146/120) (line-too-long)
    searx/webapp.py:756:0: C0301: Line too long (132/120) (line-too-long)
    searx/webapp.py:730:9: W0511: TODO, check if timezone is calculated right (fixme)
    searx/webapp.py:1:0: C0114: Missing module docstring (missing-module-docstring)
    searx/webapp.py:126:8: I1101: Module 'setproctitle' has no 'setthreadtitle' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
    searx/webapp.py:126:36: W0212: Access to a protected member _name of a client class (protected-access)
    searx/webapp.py:131:4: R1722: Consider using sys.exit() (consider-using-sys-exit)
    searx/webapp.py:141:4: R1722: Consider using sys.exit() (consider-using-sys-exit)
    searx/webapp.py:255:38: W0621: Redefining name 'request' from outer scope (line 32) (redefined-outer-name)
    searx/webapp.py:307:4: W0702: No exception type(s) specified (bare-except)
    searx/webapp.py:374:24: W0621: Redefining name 'theme' from outer scope (line 155) (redefined-outer-name)
    searx/webapp.py:420:8: R1705: Unnecessary "else" after "return" (no-else-return)
    searx/webapp.py:544:4: W0621: Redefining name 'preferences' from outer scope (line 917) (redefined-outer-name)
    searx/webapp.py:551:4: W0702: No exception type(s) specified (bare-except)
    searx/webapp.py:566:15: W0703: Catching too general exception Exception (broad-except)
    searx/webapp.py:613:4: R1705: Unnecessary "elif" after "return" (no-else-return)
    searx/webapp.py:690:8: W0621: Redefining name 'search' from outer scope (line 661) (redefined-outer-name)
    searx/webapp.py:661:0: R0914: Too many local variables (22/20) (too-many-locals)
    searx/webapp.py:674:8: R1705: Unnecessary "else" after "return" (no-else-return)
    searx/webapp.py:697:11: W0703: Catching too general exception Exception (broad-except)
    searx/webapp.py:748:4: R1705: Unnecessary "elif" after "return" (no-else-return)
    searx/webapp.py:661:0: R0911: Too many return statements (9/6) (too-many-return-statements)
    searx/webapp.py:661:0: R0912: Too many branches (29/12) (too-many-branches)
    searx/webapp.py:661:0: R0915: Too many statements (74/50) (too-many-statements)
    searx/webapp.py:931:4: W0621: Redefining name 'image_proxy' from outer scope (line 1072) (redefined-outer-name)
    searx/webapp.py:946:4: W0621: Redefining name 'stats' from outer scope (line 1132) (redefined-outer-name)
    searx/webapp.py:917:0: R0914: Too many local variables (34/20) (too-many-locals)
    searx/webapp.py:917:0: R0912: Too many branches (19/12) (too-many-branches)
    searx/webapp.py:917:0: R0915: Too many statements (65/50) (too-many-statements)
    searx/webapp.py:1063:44: W0621: Redefining name 'preferences' from outer scope (line 917) (redefined-outer-name)
    searx/webapp.py:1072:0: R0911: Too many return statements (9/6) (too-many-return-statements)
    searx/webapp.py:1151:4: C0103: Variable name "SORT_PARAMETERS" doesn't conform to '(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*)|([a-z]))$' pattern (invalid-name)
    searx/webapp.py:1297:0: R1721: Unnecessary use of a comprehension (unnecessary-comprehension)
    searx/webapp.py:1303:0: C0103: Argument name "e" doesn't conform to '(([a-z][a-zA-Z0-9_]{2,30})|(_[a-z0-9_]*))$' pattern (invalid-name)
    searx/webapp.py:1303:19: W0613: Unused argument 'e' (unused-argument)
    searx/webapp.py:1338:23: W0621: Redefining name 'app' from outer scope (line 162) (redefined-outer-name)
    searx/webapp.py:1318:0: R0903: Too few public methods (1/2) (too-few-public-methods)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-27 12:30:56 +02:00
Markus Heiser ae25362205 [pylint] webapp.py - fix all wrong-import-order messages from pylint
pylint message: wrong-import-order
  Respect PEP8 import order (standard imports first, then third-party libraries,
  then local imports).

pylint message: wrong-import-position
  Do not mix code & imports

BTW:

- only one import per line
- replace licence text by SPDX tag

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-26 21:17:19 +02:00
Markus Heiser 85364edaf0 [fix] webapp.py - running without installation is not supported
Remove extension of the sys.path (aka PYTHONPATH).  Running instance directly
from repository's folder is a relict from the early beginning in
2014 (fd651083f) and is no longer supported.

Since commit dd46629 was merged the command line 'searx-run' exists and should
be used.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-26 21:17:19 +02:00
Alexandre Flament 7c76cef746
Merge pull request #94 from return42/pylint-network
Pylint searx.network
2021-05-26 18:35:29 +02:00
Markus Heiser 97d173eda0 [mod] unsplash engine - activated by default
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 17:41:10 +02:00
Markus Heiser dc21cb5d4b [fix] unsplash engine - 'searx:result: invalid title:'
- Use result 'alt_description' as title, if not given use
  default title 'unknown'.
- Use result 'description' from unsplash as 'content'

Fix error::

    DEBUG:searx:result: invalid title: {..., 'title': None, 'content': '', 'engine': 'unsplash'}

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 17:26:58 +02:00
Markus Heiser a88e3e4fea [pylint] searx/engines/unsplash.py, add logger & norm indentation
- fix messages from pylint
- add logger and log request URL
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-25 16:45:32 +02:00
Markus Heiser 2128022f72 [coding-style] searx/network/network.py - normalized indentations
No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:48:47 +02:00
Markus Heiser 1499002ceb [coding-style] searx/network/client.py - normalized indentations
No functional change!

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:44:43 +02:00
Markus Heiser e4211da639 [pylint] searx/network/raise_for_httperror.py
No functional change!

- fix messages from pylint
- add ``global NETWORKS``
- normalized indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:40:10 +02:00
Markus Heiser 44efa911ba [pylint] searx/network/network.py & add global (NETWORKS)
No functional change!

- fix messages from pylint
- add ``global NETWORKS``

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:46 +02:00
Markus Heiser b595c482d0 [pylint] searx/network/client.py & add global (TRANSPORT_KWARGS)
No functional change!

- fix messages from pylint
- add ``global TRANSPORT_KWARGS``
- normalized python_socks imports

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:37 +02:00
Markus Heiser 8033518899 [pylint] searx/network/__init__.py & add global (THREADLOCAL)
No functional change!

- fix messages from pylint
- add ``global THREADLOCAL``
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 17:39:14 +02:00
Markus Heiser f963759ccc [fix] engine genius should not use the video template
Remove 'template' from result.  Engine genius should
not use the video template.  BTW: fix indentations

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 16:31:14 +02:00
Markus Heiser 3a71d4b175 [pylint] searx/engines/genius.py, add logger & normalized indentation
- pylint searx/engines/genius.py
- add logger and log ignored exceptions
- normalized various indentation

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24 16:19:06 +02:00
Markus Heiser 84a943f867 [enh] XPath engine - add time safe-search support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 22:26:18 +02:00
Markus Heiser 6bfe3fd033 [enh] XPath engine - add time range support
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 16:49:30 +02:00
Markus Heiser 1933577c8e [enh] XPath engine - add ISO 639-1 {lang} replacement to search-URL
BTW: remove obsolte params['query'] and not needed paging condition.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 15:05:36 +02:00
Markus Heiser 8cd544b2a6 [doc] add documentation about the XPath engine
- pylint searx/engines/xpath.py
- fix indentation of some long lines
- add logging
- add doc-strings

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-23 11:48:21 +02:00
Markus Heiser 7728e25b11 [refactor] metrics.get_reliabilities() - make code more readable
- init stat values by None
- drop round_or_none
- don't try to get percentage if base is 'None'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-22 15:17:18 +02:00
Alexandre Flament 99aaf86b73 [fix] offline engines: fix templates /stats and /preferences 2021-05-22 15:17:18 +02:00
Alexandre Flament 3014463fed [fix] metrics: processing time = total time if there is no http time
It was previsouly None

Fix /stats
2021-05-22 15:17:18 +02:00
Alexandre Flament ec83493538 [fix] offline engine: don't crash on time recording 2021-05-22 15:17:18 +02:00
Markus Heiser 6f1446d55f [pylint] searx/search/__init__.py & replace lic-text by SPDX tag
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-21 17:31:22 +02:00
Alexandre Flament 426fadccb3 [mod] remove gc.collect() after each user request 2021-05-21 17:23:18 +02:00
Alexandre Flament d1cffc55cb [fix] unit test: don't load /etc/searx/settings.yml
Add a new environment variable SEARX_DISABLE_ETC_SETTINGS
to disable loading of /etc/searx/settings.yml

unit tests:
* set SEARX_DISABLE_ETC_SETTINGS to 1
* remove SEARX_SETTINGS_PATH if it exists
2021-05-18 17:23:21 +02:00