The GSA headers that were introduced in PR #5644 unfortunately no longer
work (#6359).
The Google engines
- google.py
- google_videos.py
do not work anymore either, but we'll leave it in the code for now:
- In google.py, central functions like get_google_info(..) are provided, which
are also used by other modules.
- We will probably need a Google HTML (and video) engine again very soon.
Related:
- https://github.com/searxng/searxng/issues/6359
- https://github.com/searxng/searxng/pull/6364
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Adds a new google cse engine as an alternative to the currently broken html
google engine.
It works by generating a token which is cached for 1 hour and it only uses the
public CSE id from blackle.com. We do have two others but I think just
hardcoding one will be enough.
Will be broken in 2027 due to google deprecating it but I it will be good while
it lasts.
I found I can get up to 20 results per page. Setting safesearch and language
works and I just copied the traits from the google engine.
Also set old google engine as inactive as it is completely broken for now.
We're now shuffling the order of query arguments, as
it seems that Qwant actively blocks the query order of
SearXNG.
tgp seems to be short for "test group" (look at the XHR requests the
browser sends to "https://www.qwant.com/action/ui") - its actual value
doesn't matter, as long as it's sent and at the correct position in the
query params and doesn't change too frequently
Naver redesigned their search results page, replacing the old CSS classes
(lst_total, link_tit, total_dsc_wrap, api_txt_lines) with a new layout
using fds-web-doc-root containers and sds-comps design system.
Updated parse_general():
- Result container: fds-web-doc-root + fds-web-normal-doc-root
- Title: sds-comps-text-type-headline1 span
- URL: first external http link (with try/except guard)
- Content: sds-comps-text-type-body1 span
- Thumbnail: profile-thumbnail img src
- Added guard: skip results without title or URL
Baidu image search returns antiFlag=1 when the acjson API is called
without session cookies from image.baidu.com.
Warm up cookies with a GET to the image homepage, cache them for one
hour via EngineCache, and pass them to image search requests. Raise
SearxEngineAccessDeniedException when antiFlag is still returned.
Signed-off-by: Hu Butui <hot123tea123@gmail.com>
FindFiles.net is a specialized file search engine designed to help you search
files online with precision. Unlike traditional search engines that mainly index
web pages, FindFiles focuses on finding real files on the internet - including
PDFs, documents, archives, videos, datasets, and more. [1]
[1] https://findfiles.net
The code that reads the value of variable `x` from `embed.js`, decodes
it to ASCII and based on that sets `window["tuskheader"]` and `window["tuskkey"]`
is attached below. The only real way to figure out what this is doing is
by stepping through it with the debugger, otherwise it's almost hopeless.
```js
function fe() {
const B = pe => pe.map(_e => String.fromCharCode(_e)).join(''),
ae = window,
o = ae.x;
if (o?.length) {
const pe = o.length / 2;
for (let _e = 0; _e < pe; _e++) ae[B(o[_e])] = B(o[pe + _e]);
ae.x = void 0
}
}
```
Minimal script for testing the engine:
```py
import random
from json import loads
import requests
resp = requests.get("https://api.tusksearch.com/revcontent/embed.js")
data = loads(resp.text[6:])
def _decode(text: list[int]) -> str:
return "".join([chr(x) for x in text])
header = _decode(data[3])
value = _decode(data[4])
resp = requests.get(
"https://api.tusksearch.com/Search/Web?q=test&p=1&l=center&nextArgs=&prevArgs=",
# "https://api.tusksearch.com/Search/Image?q=test&p=1&l=center",
headers={
header: value,
'x-lon': str(random.random() * 90),
'x-lat': str(random.random() * 90),
},
)
print(resp.text)
```
Category for searching personal blogs and websites.
Useful if searching for interesting articles on a topic
rather than the mainstream Wikipedia etc. results.
The old property should still be supported for a transitional period; the
reasons for this can be seen from the discussion in [1] / the further procedure
is also discussed there.
[1] https://github.com/searxng/searxng/issues/6261
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the past, the engine option ``language_support`` was not consistently
maintained; with this patch, a ValueError is now thrown if an engine has
languages in its traits but language_support is not set to True.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Drop outdated engine attributes: supported_languages, language_aliases
Complete, normalize and document the type definitions for the engine-module and
engine-class.
For the ``engine.about`` section of the configuration, a type check is performed
based on structure ``searx.enginelib.EngineAbout``.
The property ``engine.about.language`` no longer exists; existing values have
been migrated to ``engine.language``.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
T-Online_ is a German news portal.
It gets its web results from Google, image results from Flickr and videos results
from YouTube.
For images and videos, it additionally returns result from its
news catalog. However, for pagination we have to specify the result
type (e.g. either videos from YouTube or from T-Online), so we use
flickr/youtube there instead of tonline because the tonline results
are usually irrelevant.
Add support for https://luxxle.com
Localization is not yet supported because it doesn't seem to work on their
website either, no matter which language I select, it only returns English web
results
S1Search provides various different search services, which all seem
to be somewhat based on Google and Yahoo. The site looks kinda suspicious,
but the results are fine.
You can find a list of their engines by using a subdomain finder like
https://web-toolbox.dev/en/tools/subdomain-lookup and search for `s1search.co`.
Chatnoir is an open source search engine developed by universities, based on
CommonCrawl (and others). It's uncommented by default - we don't want to
overload the universities with bot traffic that targets SearXNG (sad truth why
we can't have nice things anymore)
* [mod] template images.html: reformatted for readability (no func change)
In preparation for upcoming changes, the template is being reformatted for
better readability; no functional changes are being made.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] image results: add list of alternative formats
To test alternatives formats apply patch from below, query ``!flaticon bmw`` and
open the detail view for the image.
diff --git a/searx/engines/flaticon.py b/searx/engines/flaticon.py
index 06b6a8e25..d88388705 100644
--- a/searx/engines/flaticon.py
+++ b/searx/engines/flaticon.py
@@ -8,7 +8,7 @@ from urllib.parse import urlencode
import typing as t
-from searx.result_types import EngineResults
+from searx.result_types import EngineResults, ImageRef
if t.TYPE_CHECKING:
from searx.extended_types import SXNG_Response
@@ -61,6 +61,14 @@ def response(resp: "SXNG_Response"):
thumbnail_src=_fix_url(result["png"]),
img_src=_fix_url(result["png512"]),
author=result["team_name"],
+ formats=[
+ ImageRef(label="PNG 100x100", url="https://example.org/test.png", subtype="png"),
+ ImageRef(label="SVG", url="https://example.org/test.svg", subtype="svg+xml"),
+ ImageRef(url="https://example.org/test.jpg", subtype="jpeg"),
+ ImageRef(url="https://example.org/test.bmp", subtype="bmp"),
+ ImageRef(url="https://example.org/test.ico", subtype="x-icon"),
+ ImageRef(url="https://example.org/test.tif", subtype="tiff"),
+ ],
)
)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---------
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
e.g. when searching for "!tiger pottering github", it crashes.
not really sure why - the problem is that the HTML doesn't
really uses descriptive classes or ids, only Tailwind,
so it's very hard to select only the results HTML.
html.duckduckgo.com captchas all my IPs very fast. I figured out that using
duckduckgo.com works even if html.duckduckgo.com is captcha-ed, hence adding
support for duckduckgo.com's general web search here.
This implementation fetches the link to the first API page
(i.e. ``links.duckduckgo.com/d.js?...``) from duckduckgo.com and uses the ``n``
parameter of the API to fetch all subsequent pages.
This also means that it's not possible to immediately search for the third
page - the first and the second page would need to be loaded first.
The reason why we can't just normally use the `vqd` value is that the API URLs
require an additional parameter `dp` which seems generated at server-side, so we
can't build it ourselves and must scrape it from the HTML pages.
The initialization of the DB schema ("base schema") has so far been done on
demand, which causes race conditions with competing threads and processes.
The DDL statements for creating the "base schema" are now executed as part of
the initialization of the app.
Further improvements were made to harden the database applications:
- Wikidata & Radio-Browser engine perform their initialization only once (so far
the initialization was carried out in each thread/process).
- If multiple processes try to set DB's WAL mode when opening the DB at the same
time, this usually leads to another race condition, which is now also caught.
Related:
- https://github.com/searxng/searxng/issues/6181#issuecomment-4586705Closes: #6181
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
