mirror of
https://github.com/searxng/searxng.git
synced 2026-09-12 01:06:04 +00:00
[fix] startpage engine: language/region & time support & fix CAPTCHA
One reason for the often seen CAPTCHA of the startpage requests are the
incomplete requests SearXNG sends to startpage.com. To avoid CAPTCHA we need to
send a well formed HTTP POST request with a cookie, we need to form a request
that is identical to the request build by startpage.com itself:
- in the cookie the **region** is selected
- in the POST arguments the **language** is selected
Based on the *engine_properties* boilerplate, SearXNG's startpage engine now
implements a `_fetch_engine_properties()` function to fetch regions & languages
from startpage.com.
This patch is a complete new implementation of the request() function, reversed
engineered from the startpage.com page. The new implementation adds
- time-range support
- save-search support
to the startpage engine which has been missed in the past.
The locale code 'no_NO' from startpage does not exists and is mapped to nb-NO.
For reference see languages-subtag at iana [1], `no` is the macrolanguage::
type: language
Subtag: nb
Description: Norwegian Bokmål
Added: 2005-10-16
Suppress-Script: Latn
Macrolanguage: no
Additional hints:
- To fetch languages from startpage, this patch makes use of the
EngineProperties implemented in 7bf0d46c
- Te get Startpage's locale & language, the function get_engine_locale from
9ae409a is used.
[1] https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry
[2] https://www.w3.org/International/questions/qa-choosing-language-tags#langsubtag
Closes: https://github.com/searxng/searxng/issues/1081
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This commit is contained in:
@@ -11,9 +11,9 @@ from lxml import etree
|
||||
from httpx import HTTPError
|
||||
|
||||
from searx import settings
|
||||
from searx.data import ENGINES_LANGUAGES
|
||||
from searx.network import get as http_get
|
||||
from searx.exceptions import SearxEngineResponseException
|
||||
from searx.engines import engines
|
||||
|
||||
# a _fetch_supported_properites() for XPath engines isn't available right now
|
||||
# _brave = ENGINES_LANGUAGES['brave'].keys()
|
||||
@@ -110,9 +110,11 @@ def seznam(query, _lang):
|
||||
|
||||
def startpage(query, lang):
|
||||
# startpage autocompleter
|
||||
lui = ENGINES_LANGUAGES['startpage'].get(lang, 'english')
|
||||
engine = engines['startpage']
|
||||
_, engine_language, _ = engine.get_engine_locale(lang)
|
||||
|
||||
url = 'https://startpage.com/suggestions?{query}'
|
||||
resp = get(url.format(query=urlencode({'q': query, 'segment': 'startpage.udog', 'lui': lui})))
|
||||
resp = get(url.format(query=urlencode({'q': query, 'segment': 'startpage.udog', 'lui': engine_language})))
|
||||
data = resp.json()
|
||||
return [e['text'] for e in data.get('suggestions', []) if 'text' in e]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user