* [mod] update_engine_descriptions.py - revision of the script (#6309)
The script was a bit outdated, type hints were missing, and the strings were
converted to double quotation marks
The `descriptions.items()` are sorted to avoid assigning the references to
seen_descriptions randomly (should reduce unnecessary diffs in the future).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [data] update searx.data - update_engine_descriptions.py (#6309)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---------
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
FindFiles.net is a specialized file search engine designed to help you search
files online with precision. Unlike traditional search engines that mainly index
web pages, FindFiles focuses on finding real files on the internet - including
PDFs, documents, archives, videos, datasets, and more. [1]
[1] https://findfiles.net
The code that reads the value of variable `x` from `embed.js`, decodes
it to ASCII and based on that sets `window["tuskheader"]` and `window["tuskkey"]`
is attached below. The only real way to figure out what this is doing is
by stepping through it with the debugger, otherwise it's almost hopeless.
```js
function fe() {
const B = pe => pe.map(_e => String.fromCharCode(_e)).join(''),
ae = window,
o = ae.x;
if (o?.length) {
const pe = o.length / 2;
for (let _e = 0; _e < pe; _e++) ae[B(o[_e])] = B(o[pe + _e]);
ae.x = void 0
}
}
```
Minimal script for testing the engine:
```py
import random
from json import loads
import requests
resp = requests.get("https://api.tusksearch.com/revcontent/embed.js")
data = loads(resp.text[6:])
def _decode(text: list[int]) -> str:
return "".join([chr(x) for x in text])
header = _decode(data[3])
value = _decode(data[4])
resp = requests.get(
"https://api.tusksearch.com/Search/Web?q=test&p=1&l=center&nextArgs=&prevArgs=",
# "https://api.tusksearch.com/Search/Image?q=test&p=1&l=center",
headers={
header: value,
'x-lon': str(random.random() * 90),
'x-lat': str(random.random() * 90),
},
)
print(resp.text)
```
Category for searching personal blogs and websites.
Useful if searching for interesting articles on a topic
rather than the mainstream Wikipedia etc. results.
The old property should still be supported for a transitional period; the
reasons for this can be seen from the discussion in [1] / the further procedure
is also discussed there.
[1] https://github.com/searxng/searxng/issues/6261
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the past, the engine option ``language_support`` was not consistently
maintained; with this patch, a ValueError is now thrown if an engine has
languages in its traits but language_support is not set to True.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Drop outdated engine attributes: supported_languages, language_aliases
Complete, normalize and document the type definitions for the engine-module and
engine-class.
For the ``engine.about`` section of the configuration, a type check is performed
based on structure ``searx.enginelib.EngineAbout``.
The property ``engine.about.language`` no longer exists; existing values have
been migrated to ``engine.language``.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The results seem to be from Brave (i.e. they are exactly
the same). But it doesn't have any strict rate-limits,
so that's nice.
News support time ranges, but apart from that, unfortunately it doesn't
support any advanced features like safesearch or languages.
T-Online_ is a German news portal.
It gets its web results from Google, image results from Flickr and videos results
from YouTube.
For images and videos, it additionally returns result from its
news catalog. However, for pagination we have to specify the result
type (e.g. either videos from YouTube or from T-Online), so we use
flickr/youtube there instead of tonline because the tonline results
are usually irrelevant.
Add support for https://luxxle.com
Localization is not yet supported because it doesn't seem to work on their
website either, no matter which language I select, it only returns English web
results
RawWeb is a search engine for personal websites / blog posts.
It has its own index and the personal websites were selected
by hand. Results are quite good for what it is imo. [^1]
[^1]: https://github.com/0x2E/RawWeb.org
- adds support for https://reloado.com (german)
- as it has its own index, the results are hit or miss and mostly German,
but still worth integrating imo
Add support for https://startsiden.abcnyheter.no, a netherlandish search engine
that probably uses Google or Bing? idk it also returns English results, but
e.g. ``test`` returns mostly results from netherlands.
S1Search provides various different search services, which all seem
to be somewhat based on Google and Yahoo. The site looks kinda suspicious,
but the results are fine.
You can find a list of their engines by using a subdomain finder like
https://web-toolbox.dev/en/tools/subdomain-lookup and search for `s1search.co`.
Chatnoir is an open source search engine developed by universities, based on
CommonCrawl (and others). It's uncommented by default - we don't want to
overload the universities with bot traffic that targets SearXNG (sad truth why
we can't have nice things anymore)
* [mod] template images.html: reformatted for readability (no func change)
In preparation for upcoming changes, the template is being reformatted for
better readability; no functional changes are being made.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] image results: add list of alternative formats
To test alternatives formats apply patch from below, query ``!flaticon bmw`` and
open the detail view for the image.
diff --git a/searx/engines/flaticon.py b/searx/engines/flaticon.py
index 06b6a8e25..d88388705 100644
--- a/searx/engines/flaticon.py
+++ b/searx/engines/flaticon.py
@@ -8,7 +8,7 @@ from urllib.parse import urlencode
import typing as t
-from searx.result_types import EngineResults
+from searx.result_types import EngineResults, ImageRef
if t.TYPE_CHECKING:
from searx.extended_types import SXNG_Response
@@ -61,6 +61,14 @@ def response(resp: "SXNG_Response"):
thumbnail_src=_fix_url(result["png"]),
img_src=_fix_url(result["png512"]),
author=result["team_name"],
+ formats=[
+ ImageRef(label="PNG 100x100", url="https://example.org/test.png", subtype="png"),
+ ImageRef(label="SVG", url="https://example.org/test.svg", subtype="svg+xml"),
+ ImageRef(url="https://example.org/test.jpg", subtype="jpeg"),
+ ImageRef(url="https://example.org/test.bmp", subtype="bmp"),
+ ImageRef(url="https://example.org/test.ico", subtype="x-icon"),
+ ImageRef(url="https://example.org/test.tif", subtype="tiff"),
+ ],
)
)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---------
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
e.g. when searching for "!tiger pottering github", it crashes.
not really sure why - the problem is that the HTML doesn't
really uses descriptive classes or ids, only Tailwind,
so it's very hard to select only the results HTML.
The default Helix configuration for Python is different,
so the pylint warnings aren't shown and the formatter
re-formats files by accident when you edit an existing file.
Therefore, this commit adds `python` language configuration
to ease developing SearXNG with Helix Editor [^1].
[^1]: https://helix-editor.com
html.duckduckgo.com captchas all my IPs very fast. I figured out that using
duckduckgo.com works even if html.duckduckgo.com is captcha-ed, hence adding
support for duckduckgo.com's general web search here.
This implementation fetches the link to the first API page
(i.e. ``links.duckduckgo.com/d.js?...``) from duckduckgo.com and uses the ``n``
parameter of the API to fetch all subsequent pages.
This also means that it's not possible to immediately search for the third
page - the first and the second page would need to be loaded first.
The reason why we can't just normally use the `vqd` value is that the API URLs
require an additional parameter `dp` which seems generated at server-side, so we
can't build it ourselves and must scrape it from the HTML pages.
The initialization of the DB schema ("base schema") has so far been done on
demand, which causes race conditions with competing threads and processes.
The DDL statements for creating the "base schema" are now executed as part of
the initialization of the app.
Further improvements were made to harden the database applications:
- Wikidata & Radio-Browser engine perform their initialization only once (so far
the initialization was carried out in each thread/process).
- If multiple processes try to set DB's WAL mode when opening the DB at the same
time, this usually leads to another race condition, which is now also caught.
Related:
- https://github.com/searxng/searxng/issues/6181#issuecomment-4586705Closes: #6181
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add support for https://tiger.ch (general, news)
It is disabled and inactive by default because it's just a metasearch engine
like SearXNG is, so it's mostly useful for bypassing rate-limits on other
engines: (it has its own German index, but it's not that great) in theory it
supports different locales, but I was too lazy to implement that (I only need
German and English results anyways, which are returned by default...)
Add support for https://seek.ninja (general)
It's very slow because the engine uses Server-side events, that incrementally
send data in their HTTP response [1].
I.e. we wait for the end of the response (7+ seconds), even though the results
data arrives within a few seconds -> it's very slow, because SearXNG wants to
get the full response body before it calls the `response(resp)` method
We could use httpx-sse [2], but I'm not sure how to integrate this into SearXNG
and if it's worth it
[1] https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/
[2] https://github.com/florimondmanca/httpx-sse
The engine is using very aggressive Cloudflare blocking for
a while now, no matter if using a normal browser like Firefox
or not.
Closes: https://github.com/searxng/searxng/issues/5976
On the first page of the WEB search, there are, among other things, sections for
videos and news. The video results from these sections should not be used as
results in the WEB search of SearXNG.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
PodcastIndex.org started using a Proof-of-Work JavaScript
challenge whose results are sent as `X-Pow-*` request headers.
Although it is technically possible to re-implement the
PoW challenge in Python, it's likely impossible to maintain
because
- the actual Proof of Concept logic might change very often
- the whole idea of the Proof of Work challenge is to use
a "big" amount of resources (about 1s on my PC); so executing the challenge
would almost block all other work on the SearXNG instance
At first glance, the challenge looks very similar to what
Anubis does, because it also uses SHA256 hashes.
Exceptions in the execution of the callback must be caught / ignored and logged
on the ERROR log.
To test, apply this patch to provoke a ValueError exception::
diff --git a/searx/data/tracker_patterns.py b/searx/data/tracker_patterns.py
index ed4415bce..695ed05d2 100644
--- a/searx/data/tracker_patterns.py
+++ b/searx/data/tracker_patterns.py
@@ -114,6 +114,7 @@ class TrackerPatternsDB:
Returns bool ``True`` to use URL unchanged (``False`` to ignore URL).
If URL should be modified, the returned string is the new URL to use.
"""
+ raise ValueError("test callback exceptions")
new_url = url
parsed_new_url = urlparse(url=new_url)
Start a `make run` instance and query for example `amazon` .. have a look at the
ERROR log:
ERROR searx.result_types: filter_urls (field 'url'): ignore ValueError('test callback exceptions') from callback searx/data/tracker_patterns.py:117
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The implementation is normalized, type annotations are applied, and the results
are freed from the HTML markup (which is partially present).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- https://lite.qwant.com seems to be dead.
- The request parameters were changed to match the ones from the Qwant website.
- Qwant is now set to inactive by default due to its strict rate-limits
In the result-list, the ``number_of_results`` indicate the number of hits in the
Index, they do not indicate how many results are in the answer.
In the past, search engines such as google or ddg had an indication on the first
page of a search term of how many hits there were for this term in total in
their index.
This info was added up in SearXNG and delivered under ``number_of_results``.
Nowadays the search engines no longer indicate how many hits there are in the
index and so this field in SearXNG is also superfluous.
- https://github.com/searxng/searxng/issues/2457#issuecomment-2566181574
- https://github.com/searxng/searxng/issues/2987
- https://github.com/searxng/searxng/issues/5034
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
[mod] normalize variable name for the max number of results per request
In the past, we have used different names for the variable that specifies the
maximum number of hits in the outgoing request.
- ``page_size``
- ``number_of_results``
- ``nb_per_page``
Since *page_size* is the most accurate term and is also used in the XPath
engines, all other engines are adjusted accordingly within this
patch .. documentation adjusted accordingly.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Changes:
- Setting the "abp" query parameter causes instant blocks, it's no longer
used at Startpage
- The safesearch map changed for both the request form and the cookies. As
we were sending invalid values, that also made it easier to detect us
I've been profiling the `/preferences` endpoint using werkzeug's
`ProfilerMiddleware` (i.e. just do `app.wsgi_app = ProfilerMiddleware(app.wsgi_app)`)
and look at the outputs in the terminal when doing `make run`.
It turns out that 95%+ of the time spent were inside babel's
Locale parsing (> 700ms on my machine). That's because, when opening the settings,
we loaded the full engine traits of each engine and checked if it matches
the user-defined search language. As we have 250+ engines, and babel is
very slow when parsing Locale's, this took a very long time.
By removing this feature that shows whether the selected search language
is supported by the engine, the load time went down from 800ms to 50ms
on my machine (which is still very slow, but well, that's future work on
optimizing).
- Detect HTTP 302 responses (Google redirecting to /sorry/index
without the HTTP client following the redirect)
- Detect short HTML responses (<2000 bytes) containing "/sorry/"
links (meta-refresh or JS redirect variants)
Instances with rotating IPs can set the `suspended_times.SearxEngineCaptcha` to
0 in the search settings [1], the next request will typically use a different
outgoing IP when rotating proxies are configured
[1] https://docs.searxng.org/admin/settings/settings_search.html
Previously, `update_engine_traits.py` would fetch traits for all engines, which
is very slow and by side-effect touches engine data that are unrelated to the
engine you're currently working on.
To be faster with developing `update_engine_traits.py` supports now engine
arguments.
To test, jump into the developer environment and run the script::
$ ./manage dev.env
(dev.env)$ ./searxng_extra/update/update_engine_traits.py --help
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- add button for editing docs source on GitHub
- add links to GitHub repository and public instances (searx.space)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The readability can be improve with the following two changes:
Slightly increase the content width to make the engine list more readable::
.content {
width: 52em; /* instead of 46em */
}
To avoid word wrap in bang::
p code.literal {
text-wrap: nowrap;
}
Sugested-by: https://github.com/searxng/searxng/pull/3408#issuecomment-2094082144
This PR moves the `iframe` logic into a macro, so that `videos.html` and `general.html` both can benefit from the workaround to fix YouTube results by @return42 in https://github.com/searxng/searxng/pull/5858
It also fixes that only YouTube videos contained the closing `>` after `<iframe border="0" ...`, the regression has been caused by https://github.com/searxng/searxng/pull/5858
## Why is this change important?
Currently, the page breaks if there's any non-YouTube Iframe
Here, the page ends in the middle of the results and the footer and page number selector are not visible.
The type checker in my IDE shut down after over 500 errors / after this
patch there are still 125 criticisms, however its an improvement and a better
starting point.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The bug was introduced in commit 8769b7c6d (typification of result items); this
patch fixes the bug and also addresses the peculiarity that fields can be set
but contain no *usable* value:
If a field is set (exists) but contains an empty string or the value ``None``,
it is also considered *not set*. This also ensures that an integer 0 is
evaluated *as set*!
Co-Authored: Markus Heiser <markus.heiser@darmarit.de>
Notes:
- Safesearch doesn't seem to work properly?
- In theory multiple languages are supported, but even in the web UI, they don't work properly
- Possibly, we could cache the request hashes (h query parameter), I'm not sure if it ever changes
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
<!-- If you are running on master branch using git execute this command
in order to fetch the latest commit ID:
```
git log -1
```
If you are using searxng-docker then look at the bottom of the SearXNG page
and check for the version after "Powered by SearXNG"
_Replace this placeholder with a meaningful and precise description of the bug._
Please also stipulate if you are using a forked version of SearXNG and
include a link to the fork source code.
<!-- FILL IN THESE FIELDS .. and delete the comments after reading.
Use Markdown for formatting -> https://www.markdowntools.io/cheat-sheet
-->
**How did you install SearXNG?**
<!-- Did you install SearXNG using the official wiki or using searxng-docker
or manually by executing the searx/webapp.py file? -->
**What happened?**
<!-- A clear and concise description of what the bug is. -->
**How To Reproduce**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->
### How To Reproduce?
<!-- How can we reproduce this issue? (as minimally and as precisely as
possible) -->
### Expected behavior
**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->
**Screenshots & Logs**
### Screenshots & Logs
<!-- If applicable, add screenshots, logs to help explain your problem. -->
**Additional context**
### Version of SearXNG
<!-- Commit number if you are using on master branch and stipulate if you forked
SearXNG -->
<!-- Look at the bottom of the SearXNG page and check for the version after
"Powered by SearXNG" If you are using a forked version of SearXNG include a
link to the fork source code. -->
### How did you install SearXNG?
<!-- Did you install SearXNG using the official documentation or using
searxng-docker? -->
### Additional context
<!-- Add any other context about the problem here. -->
- [ ] I read the [AI Policy](https://github.com/searxng/searxng/blob/master/AI_POLICY.rst) and hereby confirm that this issue conforms with the policy.
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->
**Working URL to the engine**
<!-- Please check if the engine is responding correctly before submitting it. -->
<!-- FILL IN THESE FIELDS .. and delete the comments after reading.
**Why do you want to add this engine?**
<!-- What's special about this engine? Is it open source or libre? -->
Use Markdown for formatting -> https://www.markdowntools.io/cheat-sheet
-->
**Features of this engine**
<!-- Features of this engine: Doesn't track its users, fast, easy to integrate, ... -->
### Working URL to the engine
**How can SearXNG fetch the information from this engine?**
<!-- List API URL, example code (using the correct markdown) and more
that could be useful for the developers in order to implement this engine.
If you don't know what to write, let this part blank. -->
<!-- Please check if the engine is responding correctly before submitting -->
**Applicable category of this engine**
<!-- Where should this new engine fit in SearXNG? Current categories in SearXNG:
general, files, images, it, map, music, news, science, social media and videos.
You can add multiple categories at the same time. -->
### Why do you want to add this engine?
**Additional context**
<!-- Add any other context about this engine here. -->
<!-- What's special about this engine? -->
- [ ] I read the [AI Policy](https://github.com/searxng/searxng/blob/master/AI_POLICY.rst) and hereby confirm that this issue conforms with the policy.
### Features of this engine
<!-- Features of this engine: Serves special content, is fast, is easy to
integrate, ... ? -->
### How can SearXNG fetch results from this engine?
<!-- List API URL, example code and more that could be useful for the developers
in order to implement this engine. If you don't know what to write, let
this part blank. -->
### Applicable category of this engine
<!-- Where should this new engine fit in SearXNG? Current categories in
SearXNG: general, files, images, it, map, music, news, science, social
media and videos. -->
### Additional context
<!-- Add any other context about the problem here. -->
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->
**Is your feature request related to a problem? Please describe.**
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
_Replace this placeholder with a concise description of the feature._
<!-- FILL IN THESE FIELDS .. and delete the comments after reading.
Use Markdown for formatting -> https://www.markdowntools.io/cheat-sheet
-->
### Is your feature request related to a problem?
<!-- A clear and concise description of what the problem is. Ex. I'm always
frustrated when [...] -->
### Describe the solution you'd like
**Describe the solution you'd like**
<!-- A clear and concise description of what you want to happen. -->
**Describe alternatives you've considered**
### Describe alternatives you've considered
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
**Additional context**
<!-- Add any other context or screenshots about the feature request here. -->
### Code of Conduct
- [ ] I read the [AI Policy](https://github.com/searxng/searxng/blob/master/AI_POLICY.rst) and hereby confirm that this issue conforms with the policy.
- [ ] I hereby confirm that I have not used any AI tools for creating this PR.
- [ ] I have used AI tools for working on the changes in this pull request and will attach a list of all AI tools I used and how I used them. I hereby confirm that I haven't used any other tools than the ones I mention below.
### What does this PR do?
<!-- Explain the motivation and changes in your pull request. -->
### How to test this PR locally?
<!-- Commands to run the tests or instructions to test the changes. Are there
any edge cases (environment, language, or other contexts) to take into
account? -->
### Related issues
<!--
Closes: #234
-->
### Code of Conduct
<!-- ⚠️ Bad AI drivers will be denounced: People who produce bad contributions
that are clearly AI (slop) will be blocked for all future contributions.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
