Commit Graph

5024 Commits

Author SHA1 Message Date
Markus Heiser 420e3caba3 [mod] enable Qwant engines (web, news, images, video) by default
Qwant is a fast and reliable search engine and AFAIK there is no CAPTCHA.  Let
us enable Qwant engines by default.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14 09:47:32 +02:00
Markus Heiser ef6e1bd6b9 [fix] Qwant engines - implement API v3 and add 'quant videos'
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.

This implementation is used by different qwant engines in the settings.yml::

  - name: qwant
    categories: general
    ...
  - name: qwant news
    categories: news
    ...
  - name: qwant images
    categories: images
    ...
  - name: qwant videos
    categories: videos
    ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14 09:47:32 +02:00
Alexandre Flament ff180cf2db
Merge pull request #203 from return42/drop-dogpile
[fix] drop engine dogpile
2021-07-13 15:58:37 +02:00
Markus Heiser 2767046065 [fix] drop engine dogpile
The engine was added in commit a4b07460 but now it shows new issues [1].

In the 90'th of the last century, dogpile had its own WEB index, but nowadays it
is a meta-search engine [2]

  Powered by technology, Dogpile returns all the best results from leading
  search engines including Google and Yahoo!

Using dogpile as an engine in SearXNG needs more investigation, a XPath solution
like we have is not enough.   It is questionable whether it still makes sense to
investigate more into a meta-search engine with a ReCAPTCHA in front.

With this patch the dogpile engine is removed

[1] https://github.com/searxng/searxng/issues/202
[2] https://www.dogpile.com/support/aboutus

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-13 14:42:09 +02:00
Alexandre Flament 4f09c7ba0c
Merge pull request #201 from return42/drop-torrentz
[drop] engine torrentz: torrentz2.eu and torrentz2.is are offline
2021-07-12 08:22:49 +02:00
Markus Heiser 513c73a309 [drop] engine torrentz: torrentz2.eu and torrentz2.is are offline
[1] https://torrentfreak.com/torrentz2-eu-domain-suspended-by-registry-on-public-prosecutors-order-200628/

Suggested-by: @rasos https://github.com/searx/searx/issues/1875#issuecomment-877755872
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-11 13:24:33 +02:00
Markus Heiser 438357a5b8
Merge pull request #200 from unixfox/patch-1
Fix google images
2021-07-10 16:17:21 +00:00
Émilien Devos d9d9bd720d
Fix google images
Proposed fix in https://github.com/searx/searx/pull/2115#issuecomment-876716010
2021-07-10 14:09:29 +00:00
Markus Heiser c72cca69de
Merge pull request #198 from return42/gpodder.net
[enh] add gpodder.net (JSON)
2021-07-03 20:46:51 +00:00
Markus Heiser 373259e9bd
Merge pull request #197 from return42/woxikon
Add woxikon.de synonyme (Xpath)
2021-07-03 20:31:46 +00:00
Allen 3b2e0a93b7 [enh] add gpodder.net (JSON)
Engine just for Podcasts.  An API which returns Podcasts and their Info like:
website, author etc.

Upstream query example: https://gpodder.net/search.json?q=linux
2021-07-03 22:30:20 +02:00
Markus Heiser d5eedf1b61 [yamllint] fix some warning issues reported by yamllint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 21:55:36 +02:00
Allen 9648d71ad8 [enh] add woxikon.de synonyme (Xpath)
Added synonyme.woxikon.de using the xpath engine.  Adds a site which returns
word synonyms although just in German.

Depending on the query not all synonyms are shown because of not the best xpath
selection.  But should do the job just fine.

Upstream example query: https://synonyme.woxikon.de/synonyme/test.php
2021-07-03 21:51:26 +02:00
Markus Heiser bb98a07b50
Merge pull request #196 from return42/fix-brave
[fix] brave engine: shows descriptions with their correct URLs
2021-07-03 19:44:48 +00:00
Markus Heiser 9d97b2e854 [fix] brave engine: shows descriptions with their correct URLs
BTW add about section to the YAML configuration

It now shows descriptions with their correct URLs when there are videos in the
search results, pulling content_xpath from snippet-description instead of
snippet-content.

Suggested-by: @eagle-dogtooth https://github.com/searx/searx/issues/2857#issuecomment-869119968
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 20:57:55 +02:00
Markus Heiser 72a55f68c8
Merge pull request #191 from searxng/dependabot/pip/master/transifex-client-0.14.3
Bump transifex-client from 0.14.2 to 0.14.3
2021-07-03 18:03:09 +00:00
Markus Heiser debcbab561 [fix] add missing transifex configuration (./tx/config)
BTW: update the documentation about transifex usage

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 19:31:31 +02:00
Markus Heiser b558c5aa36
Merge pull request #194 from return42/fix-misc
[fix] some minor issues in Makefile and shell scripts
2021-07-03 16:40:19 +00:00
Markus Heiser 129ed4c9fa [mod] add Makfile target lxc.clean which deletes ./lxc-env
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 18:25:46 +02:00
Markus Heiser b3ff510e10 [fix] pacman --noprogressbar in shell scripts
From pacman's man page:

  --noprogressbar
    Do not show a progress bar when downloading files. This can be useful for
    scripts that call pacman and capture the output.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 18:24:55 +02:00
Markus Heiser 73fc6499d8 [fix] utils/lib.sh: in a pipe, do not print ASCII escape code
In 2a12637 an ASCII escape code was added, such escape codes should not be
written to pipes (when the output is not a terminal).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 18:24:49 +02:00
Markus Heiser 7e8d1f9935
Merge pull request #178 from return42/whaleslide.com
[enh] Add WhaleSlide engine (JSON)
2021-07-03 16:22:47 +00:00
dependabot[bot] 0c256b378b
Bump transifex-client from 0.14.2 to 0.14.3
Bumps [transifex-client](https://github.com/transifex/transifex-client) from 0.14.2 to 0.14.3.
- [Release notes](https://github.com/transifex/transifex-client/releases)
- [Commits](https://github.com/transifex/transifex-client/compare/0.14.2...0.14.3)

---
updated-dependencies:
- dependency-name: transifex-client
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-03 16:19:20 +00:00
Markus Heiser e153ad131c
Merge pull request #192 from searxng/dependabot/pip/master/splinter-0.15.0
Bump splinter from 0.14.0 to 0.15.0
2021-07-03 16:18:39 +00:00
dependabot[bot] f255c8a101
Bump splinter from 0.14.0 to 0.15.0
Bumps [splinter](https://github.com/cobrateam/splinter) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/cobrateam/splinter/releases)
- [Changelog](https://github.com/cobrateam/splinter/blob/master/docs/news.rst)
- [Commits](https://github.com/cobrateam/splinter/compare/0.14.0...0.15.0)

---
updated-dependencies:
- dependency-name: splinter
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-03 16:09:46 +00:00
Markus Heiser 6ae8ae6c7b
Merge pull request #190 from searxng/dependabot/pip/master/pylint-2.9.3
Bump pylint from 2.8.3 to 2.9.3
2021-07-03 16:08:42 +00:00
dependabot[bot] 19e2b72367 Bump pylint from 2.8.3 to 2.9.3
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.8.3 to 2.9.3.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.8.3...v2.9.3)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-07-03 17:54:17 +02:00
Markus Heiser 24f2376c11 [pylint] prepare for pylint v2.9.3 / fix some (new) pylint issues
Upgrade from pylint v2.8.3 to 2.9.3 raise some new issues::

  searx/search/checker/__main__.py:37:26: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
  searx/search/checker/__main__.py:38:26: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
  searx/search/processors/__init__.py:20:0: R0402: Use 'from searx import engines' instead (consider-using-from-import)
  searx/preferences.py:182:19: C0207: Use data.split('-', maxsplit=1)[0] instead (use-maxsplit-arg)
  searx/preferences.py:506:15: R1733: Unnecessary dictionary index lookup, use 'user_setting' instead (unnecessary-dict-index-lookup)
  searx/webapp.py:436:0: C0206: Consider iterating with .items() (consider-using-dict-items)
  searx/webapp.py:950:4: C0206: Consider iterating with .items() (consider-using-dict-items)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03 17:54:08 +02:00
Alexandre Flament 961dd287a1
Merge pull request #185 from searxng/update_data_update_external_bangs.py
Update searx.data - update_external_bangs.py
2021-07-01 18:43:18 +02:00
Alexandre Flament 508195a850
Merge pull request #186 from searxng/update_data_update_wikidata_units.py
Update searx.data - update_wikidata_units.py
2021-07-01 18:41:54 +02:00
Alexandre Flament 89a27f4f96
Merge pull request #187 from searxng/update_data_update_ahmia_blacklist.py
Update searx.data - update_ahmia_blacklist.py
2021-07-01 18:41:43 +02:00
Alexandre Flament 1446f5a582
Merge pull request #188 from searxng/update_data_update_firefox_version.py
Update searx.data - update_firefox_version.py
2021-07-01 18:41:31 +02:00
dalf 973d507410 Update searx.data - update_ahmia_blacklist.py 2021-07-01 06:22:48 +00:00
dalf f6a8a5d017 Update searx.data - update_firefox_version.py 2021-07-01 06:22:48 +00:00
dalf 77a432cc95 Update searx.data - update_wikidata_units.py 2021-07-01 06:22:43 +00:00
dalf 726ad776a3 Update searx.data - update_external_bangs.py 2021-07-01 06:22:24 +00:00
Alexandre Flament 30e4a2a224
Merge pull request #174 from searxng/static_script
[mod] add make targets to manage the build files in the /searx/static directory
2021-06-28 10:53:32 +02:00
Alexandre Flament d29094579b [build] /static 2021-06-28 10:39:10 +02:00
Markus Heiser 76a066d4a3 [mod] themes: remove banner generated by grunt
The banner is useless and responsible for unwanted diff.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-28 10:38:44 +02:00
Markus Heiser 5a979ff987 [docs] describe the usage of the static.build.* scripts
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-28 10:22:06 +02:00
Alexandre Flament 7308709b8b
Merge pull request #179 from return42/brave
[enh] add Brave-Search engine (XPath)
2021-06-28 09:23:25 +02:00
Alexandre Flament b2da675e27
Merge pull request #180 from return42/unicode-url
[fix] decoding of saved preferences in the URL
2021-06-28 08:47:25 +02:00
Markus Heiser 2988ef2ca5
Merge pull request #182 from return42/fix-.config.sh
[fix] .config.sh: typo SEARX_INTERNAL_URL --> SEARX_INTERNAL_HTTP
2021-06-27 11:48:43 +00:00
Markus Heiser cc11951885 [fix] .config.sh: typo SEARX_INTERNAL_URL --> SEARX_INTERNAL_HTTP
There is a typo in .config.sh that causes confusion more and more often.
SEARX_INTERNAL_HTTP should be the correct name of the environment variable.
First mentioned in [1] and also discussed in [2].

[1] https://github.com/searx/searx/pull/2273
[2] https://github.com/searx/searx/discussions/2863

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-27 13:24:09 +02:00
Markus Heiser 17fde80ece
Merge pull request #181 from return42/fix-lib
[fix] utils/lib.sh: prefix_stdout show cursor (ANSI escape \e[?25h)
2021-06-27 10:13:12 +00:00
Markus Heiser 2a12637f59 [fix] utils/lib.sh: prefix_stdout show cursor (ANSI escape \e[?25h)
Some piped commands hide the cursor, show cursory when the stream ends.

Most often this is a bug of the command which piped.  The command should not
hide the cursor when it writes to a pipe.  I have seen this bug with the package
manager (pacman) from ArchLinux.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-27 11:48:57 +02:00
Markus Heiser 6787e5a36b [fix] decoding of saved preferences in the URL
To compress saved preferences in the URL was introduced in 5f758b2d3 and
slightly fixed in 8f4401462.  But the main fail was not fixed; The decompress
function returns a binary string and this binary should first be decoded to a
string before it is passed to urllib.parse_qs.

BTW: revert the hot-fix from 5973491

Related-to: https://github.com/searxng/searxng/issues/166
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 17:14:13 +02:00
Markus Heiser e0365d987b [enh] add Brave-Search engine (XPath)
Add https://search.brave.com which was requested by @kaonashi696 [1].
This patch was suggested by @eagle-dogtooth [2].

[1] https://github.com/searx/searx/issues/2857
[2] https://github.com/searx/searx/issues/2857#issuecomment-867199241

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 15:54:45 +02:00
Allen e2094cb226 [enh] Add WhaleSlide engine (JSON)
Add WhaleSlide engine [1] / merged from [2]

[1] https://whaleslide.com
[2] https://github.com/searx/searx/pull/2861
2021-06-26 15:28:44 +02:00
Markus Heiser 25b6309cf2 [mod] move functions from utils/manage_static.sh to ./manage script
The functions:

- static.build.commit
- static.build.commit.drop
- static.build.restore

are imported into the ./manage script.  To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:45:25 +02:00