jandieman
/
searxng
mirror of https://github.com/searxng/searxng.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
7,859 Commits 6 Branches 17 Tags 136 MiB
Commit Graph

1642 Commits

Author SHA1 Message Date
Markus Heiser 44392bd436 [mod] improve implementation of presearch engine 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-11-27 14:16:42 +01:00
Bnyro 23582aac5c [feat] implementation of presearch engine 2023-11-27 14:16:42 +01:00
Bnyro c3cc24be12 [feat] engine: implementation of destatis 2023-11-27 13:54:48 +01:00
Émilien (perso) 4280318fc5 fixing results parsing brave 2023-10-13 11:47:30 +02:00
Hackurei efada7cba2 [fix] hackernews keyerror problem 2023-10-13 08:16:47 +02:00
Hackurei af071121de [fix] imgur - incorrect wikidata id 2023-10-12 09:14:00 +02:00
Markus Heiser 14323d683f [fix] ddg-lite & ddg-extra: don't send empty vqd value 
DDG's bot detection is sensitive to the vqd value.  For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.

If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).

Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.

Related:

- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-12 08:52:28 +02:00
Markus Heiser 3388441917 [fix] ddg-lite vqd value: some search terms do not have a vqd value 
Some search terms do not have results and therefore no vqd value

BTW: remove a leftover from 9197efa

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-10 09:12:30 +02:00
Markus Heiser 9197efa2a7 [fix] duckduckgo lite engine: set HTTP header 'Referer' 
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].

From reverse engineering:

- The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite
  does not like this referer anymore!

- The 'Referer' header is only set on second and follow up pages but not on the
  first page

- The vqd value is not needed on the first page, the ddg-lite client sets this
  value only on follow up pages / this can help to reduce the vqd requests from
  SearXNG.

Related to 'Referer' header & ddg requests:

[1] https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081

Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-10 08:40:53 +02:00
Bnyro fa5b2a7948 [mod] yacy: use official instance by default and fix crashes 2023-10-09 20:50:24 +02:00
Hackurei ff78b1a902 [feat] implement hackernews engine - news.ycombinator.com 2023-10-09 14:00:04 +02:00
Aine 213cb74378 [fix] matrixrooms add proper MRS integration 
Related:

- https://github.com/searxng/searxng/issues/2918
 2023-10-09 13:25:13 +02:00
Bnyro 48cb58bd2e [feat] duckduckgo: support for videos and news 2023-10-09 06:53:43 +02:00
Bnyro c3ab49cd90 [fix] kickass: crash when no results 2023-10-07 11:48:23 +02:00
Bnyro f22daf8b47 [mod] piped: always show video length if available 2023-10-07 11:45:46 +02:00
Bnyro ce270961e8 [feat] engine: implementation of mastodon 2023-10-06 10:58:23 +02:00
Markus Heiser fd1422a670 [mod] engine - simplify region & lang handling, make filters configurable 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-05 10:55:08 +02:00
Bnyro 3e2ae756f0 [feat] engine: implementation of radio-browser.info 2023-10-05 10:55:08 +02:00
Jinyuan Huang e509cb7c45 [typo] solved a typo in yahoo error message. 2023-10-01 08:29:06 +02:00
Jinyuan Huang d4d9f2073e [fix] Bug: Yahoo results for simplified Chinese search sometimes have the first character cut off #2866 
Co-authored-by: Blair Noctis <n@sail.ng>
 2023-10-01 08:29:06 +02:00
Bnyro fe9386b58d [fix] emojipedia: fix engine 2023-10-01 08:19:45 +02:00
Markus Heiser 32a4ea350e [fix] Revision of the Bing engines 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-01 08:01:38 +02:00
jazzzooo 079636c079 [fix] engine - bing fix search, pagination, remove safesearch 2023-10-01 08:01:38 +02:00
Bnyro 5ce1792432 [feat] engine: implementation of pinterest 2023-09-30 15:01:45 +02:00
Bnyro 6096457e4d [fix] matrixrooms.info: pagination not working properly 2023-09-30 14:51:07 +02:00
Markus Heiser e1a8b8189f [fix] engine - moviepilot instead of thumbnail use img_src 
Instead of thumbnail use img_src in the result item, otherwise the "movies"
categories looks clunky.

Related:

- b4e0d2eedc (r128785388)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-30 11:29:19 +02:00
Bnyro 159629c588 [mod] tagesschau: add option to only use tagesschau urls 2023-09-30 11:00:11 +02:00
Bnyro 2ca60a19fc [feat] engine: implementation of matrixrooms.info 2023-09-30 09:09:23 +02:00
Bnyro fc4a20f734 [mod] add movies category for tmdb, imdb and moviepilot 2023-09-29 22:37:51 +02:00
jazzzooo e37d775fa2 [fix] engine - currency fix and simplify 2023-09-28 08:29:38 +02:00
Jinyuan Huang ae28d429c9 [fix] bilibili new api used 2023-09-28 08:24:51 +02:00
jazzzooo 1a66d74673 [fix] engine - kickass update url, fix parsing, use multiple mirrors 2023-09-27 10:19:41 +02:00
Markus Heiser b428ccc5a0 [fix] engine brave - fetch traits (modified settings menu) 
Brave has changed it settings menu fundamental.  Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:

  https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-27 09:12:38 +02:00
Markus Heiser 3a456b1282 [fix] engine annas archive - fetch traits (modified xpath selectors) 
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-27 09:12:38 +02:00
Émilien (perso) 1851f27154
[mod] remove twitter (#2843) 2023-09-24 08:32:53 +00:00
Bnyro 75c9de02d1 [feat] engine: implementation of imgur 2023-09-22 20:50:53 +02:00
Bnyro fb72f71f0a [fix] internet archive scholar: crash when there's no title 2023-09-22 18:49:39 +02:00
Markus Heiser 71358e9c67 Revert "[fix] engine - duckduckgo vqd edge-case" 
This reverts commit 102502a4f0.
 2023-09-22 09:31:25 +02:00
Bnyro 51236ae47a [feat] engine: implementation of chefkoch.de 2023-09-21 17:23:59 +02:00
jazzzooo 8bcca0e620 [fix] engine - brave don't show ads 2023-09-21 16:55:39 +02:00
jazzzooo b729542a66 [fix] engine - google images error when no results 2023-09-21 16:38:37 +02:00
Bnyro cc2e0537a3 [feat] engine: implementation of google icons/material design icons 2023-09-21 15:16:49 +02:00
Bnyro c999cfb422 [feat] engine: implementation of wallhaven 2023-09-21 14:25:43 +02:00
jazzzooo 102502a4f0 [fix] engine - duckduckgo vqd edge-case 2023-09-20 20:05:06 +02:00
Markus Heiser 043dcbf7c5 [fix] engine qwant (web-lite) - ignore advertising adds 
Closes: https://github.com/searxng/searxng/issues/2812
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-19 17:06:56 +02:00
Émilien (perso) ad725ce7d7
wikipedia wikidata infobox + disable wikisource (#2806) 
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-19 10:31:02 +02:00
Bnyro efd3a2d6d1 [feat] engine: implementation of internet archive scholar 2023-09-18 18:12:00 +02:00
jazzzooo 223b3487c3 [fix] spelling 2023-09-18 16:20:27 +02:00
Markus Heiser a9b6963971 [fix] engine - qwant delivers only 5 pages maximum 
all qwant engines (incl qwant-lite) delivers only 5 pages maximum

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-18 01:43:43 +02:00
jazzzooo da1446c5ed [fix] engine - qwant wrong error type 2023-09-18 01:43:43 +02:00
Markus Heiser 7398d525c8 [fix] qwant: subsequent fix of commit d9dbcedeb 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-17 17:52:56 +02:00
Markus Heiser d9dbcedeb6 [feat] implementation of qwant lite for web search 
Related: https://github.com/searxng/searxng/issues/2719
Replace: https://github.com/searxng/searxng/pull/2748
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-17 16:53:25 +02:00
Bnyro b4e0d2eedc [feat] engine: implemenation of moviepilot (de) 2023-09-17 14:30:56 +02:00
jazzzooo 7dfcc3386e [fix] tagesschau videos 2023-09-16 18:40:26 +02:00
jazzzooo ec540a967a [fix] brave.videos 2023-09-15 22:00:09 +02:00
jazzzooo 27477f51fd [fix] brave.news 2023-09-15 22:00:09 +02:00
Justas Zabulionis 41ef73ca3a [fix] rumble redirect 2023-09-14 19:21:21 +02:00
Justas Zabulionis be888810ba [fix] pubmed content being None 2023-09-14 18:40:15 +02:00
Justas Zabulionis 92d39de410 [fix] solidtorrents redirects 2023-09-14 18:03:21 +02:00
Justas Zabulionis cf8a6cf6db [fix] solidtorrents pagination 2023-09-14 18:03:21 +02:00
Justas Zabulionis 8172f89075 [fix] solidtorrents 2023-09-14 18:03:21 +02:00
jazzzooo 74600c028d [fix] engine - Crossref 
Crossref was broken on result types journal-issue and component .. The old code
had lots of assumptions, and broke during parsing.  Now the assumptions are more
explicit and checked them with the API.
 2023-09-14 17:39:23 +02:00
Bnyro 3568a3cafb [feat] odysee: implement fetch_traits for language support 2023-09-13 21:41:33 +02:00
Bnyro 09c61dabc9 [mod] odysee: time range support 2023-09-13 21:41:33 +02:00
jazzzooo b98907e91f [fix] engine - piped.music incorrect timestamps 2023-09-13 21:39:37 +02:00
jazzzooo 6039dbf211 [fix] engine - invidious thumbnails 2023-09-13 11:37:42 +02:00
jazzzooo b2fd6304bf [fix] engine - openstreetmap currency rendering 2023-09-13 10:56:52 +02:00
jazzzooo 54a3e03b45 [fix] engine - openstreetmap currency matching 2023-09-12 20:57:05 +02:00
Bnyro 64d9587ac8 [feat] new engine: svgrepo 2023-09-12 20:38:36 +02:00
jazzzooo b189578b6b [fix] engine - brave 2023-09-12 11:31:43 +02:00
Bnyro f182abd6f8 [mod] library of congress: fix engine 2023-09-11 19:42:31 +02:00
Bnyro e73a6f5d14 [fix] engine deviantart: review of the result-scrapper 
The deviantart site changed and hence deviantart is currently unusable.
 2023-09-11 13:22:36 +02:00
Alexandre Flament d07c006aed Replace chompjs with pure Python code 
The new implementation is good enough for the current usage (brave)
 2023-09-09 13:02:36 +02:00
Bnyro 9e83c0dedc [feat] engine: implementation of Yummly 
Co-authored-by: Markus Heiser <markus.heiser@damarit.de>
 2023-09-08 11:47:13 +02:00
Bnyro a3d7e9c285 [mod] utils.py: add markdown_to_text helper function 2023-09-08 11:47:13 +02:00
Hackurei 1f21ac7d62 [feat] engine: implementation of bilibili https://www.bilibili.com 2023-09-05 22:53:03 +02:00
Markus Heiser 696c35d2c3 [fix] engine - duckduckgo_images / determination of vqd value incorrect 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-05 22:24:51 +02:00
bonswouar 4fb26cd96d [fix] engine duckduckgo weather api changes 2023-09-05 16:55:00 +02:00
Markus Heiser 01be9e0e20 [fix] engine: wikicommons - don't quoute ':|' in URL parameters 
From [1]: It seems to be because of [2] For some reason it gets url encoded
twice, resulting in

- ``filetype%253Abitmap%257Cdrawing+birds`` instead of
- ``filetype:bitmap%7Cdrawing+birds``

[1] https://github.com/searxng/searxng/issues/2707
[2] https://github.com/searxng/searxng/blob/master/searx/engines/wikicommons.py#L43

Closes: #2707
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-03 18:45:39 +02:00
Markus Heiser 4f8895c6de [fix] follow-up of 4da7003ae / add missing review from @Bnyro 
[1] https://github.com/searxng/searxng/pull/2656#pullrequestreview-1607956209

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-02 09:26:45 +02:00
Hackurei 4da7003ae0 [feat] engine: implementation of odysee 2023-09-02 09:14:12 +02:00
Bnyro 9c4e9d3814 [feat] implementation of Wikimedia commons for images 2023-09-01 18:39:24 +02:00
Alexandre Flament faa4280e1a [mod] bing: resolve redirect without additional requests 
Remove the usage of searx.network.multi_requests
The results from Bing contains the target URL encoded in base64
See the u parameter, remove the first two character "a1", and done.

Also add a comment the check of the result_len / pageno
( from https://github.com/searx/searx/pull/1387 )
 2023-08-29 07:39:06 +02:00
Markus Heiser b0d2cd5ca9 [doc] add documentation of Mwmbl engine & autocompleter 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-27 17:25:26 +02:00
Bnyro 19300a5659 [mod] engine mwmbl: add link to official api docs 2023-08-27 17:25:26 +02:00
Alexandre Flament e16c007c22 [fix] openstreetmap engine 
It seems there is an API change:
extratags can be either a dictionnary or None.

This commit avoid crash when extratags is None

Test query "!osm gare du nord"
 2023-08-27 11:49:16 +02:00
Markus Heiser 0647f83a3e [fix] google engine: don't overspecify the search query to Google 
The method EngineTraits.get_region(..) returns engine's region string
that **best fits** to SearXNG's locale.  This means it returns a
region (country) if only a language is set in the locale.  By example the method
returns for a locale tag `es` a region `ES`.

Google's search parameter `cr` restricts search results to documents originating
in a particular country / in case of a locale tag (language) as described above,
this argument should be unset in the query send to Google.

Closes: https://github.com/searxng/searxng/issues/2672
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-26 07:47:07 +02:00
Markus Heiser 4b42644579 [fix] engine google_video: google has changed the layout of the rsponse 
Closes: https://github.com/searxng/searxng/issues/2664
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-22 08:34:04 +02:00
Bnyro c59ae91b76 [feat] engine: implementation of mwmbl 2023-08-19 18:23:42 +02:00
Markus Heiser c741fc6f00 [mod] currency_convert: support for showing the answer source url 
Show URL of the ddg-search page, not the URL of a (generic) Javascript.  The
latter one is not usefull for the user.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-18 19:07:14 +02:00
Markus Heiser e2744520f8 [mod] google: support for showing the answer source url 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-18 19:07:14 +02:00
Bnyro 5ec7df3480 [mod] engine duckduckgo definitions: support for answer source 2023-08-18 19:07:14 +02:00
Bnyro 64bc98b5fb [mod] brave: support for showing the answer source url 2023-08-18 19:07:14 +02:00
Markus Heiser 9100a48541 [mod] improve seekr engines and add documentation 
Tis patch adds some more fields to the result items and changed paging to the
``nextResultSet`` given in seekr's JSON response.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-15 16:17:42 +02:00
Bnyro 2bab658d39 [feat] engine: implementation of seekr for news, images and videos 2023-08-15 16:17:42 +02:00
Bnyro e25d1c7288 [feat] engine: implementation of German news, Tagesschau 
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-10 20:27:54 +02:00
Bnyro 834e1c3f12 [mod] engine lemmy: increase thumbnail quality to align with theme 2023-08-10 12:58:40 +02:00
Markus Heiser c381fc001f [mod] settings: remove lemmy from categ 'general' & enable by default 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-10 12:58:40 +02:00
Markus Heiser fda111c0c9 [mod] engine lemmy: add more info fields to the result items 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-08-10 12:58:40 +02:00
Bnyro 224f2250ae [feat] engine: support for lemmy communities, posts, comments and users 2023-08-10 12:58:40 +02:00