Commit Graph

2188 Commits

Author SHA1 Message Date
Bnyro
fe5bdd1bf2 [feat] magnific: solve captcha to bypass botblocking
Magnific now employs a very simple to solve CAPTCHA before
all API requests (essentially summing up some numbers).

Python script for testing:
```python
import re
from lxml import html
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:156.0) Gecko/20100101 Firefox/156.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.9',
    'Sec-GPC': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'same-origin',
    'Priority': 'u=0, i',
    'Pragma': 'no-cache',
    'Cache-Control': 'no-cache',
}

resp = requests.get(
    "https://www.magnific.com/api/regular/search?term=tree&filters%5Bai-generated%5D%5Bexcluded%5D=1&page=1&locale=en&filters%5Blicense%5D=free",
    headers=headers,
)
cookies = resp.cookies
print(resp.text)

_NUMBER_RE = re.compile("\d+")
_BM_VERIFY = re.compile(r"\"bm-verify\":\s*\"(.*?)\"")

doc = html.fromstring(resp.text)
script = doc.xpath("//script")[0].text
numbers = [int(m) for m in _NUMBER_RE.findall(script)]
bm_verify = _BM_VERIFY.search(resp.text)
if not bm_verify:
    exit(1)
bm_verify = bm_verify.group(1)
print("bm-verify: " + bm_verify)

solution = sum(numbers)
print(f"sum({numbers}) = {solution}")

resp = requests.post(
    "https://www.magnific.com/_sec/verify?provider=interstitial",
    json={"bm-verify": bm_verify, "pow": solution},
    headers=headers,
    cookies=cookies,
)

resp = requests.get(
    "https://www.magnific.com/api/regular/search?term=bird&filters%5Bai-generated%5D%5Bexcluded%5D=1&page=1&locale=en&filters%5Blicense%5D=free",
    headers=headers,
    cookies=resp.cookies,
)
print(resp.text)
```
2026-09-19 11:31:00 +02:00
Brock Vojkovic
367fb6537c [fix] engines: tusksearch cors request headers (#6758)
Added the ``Origin`` and ``Sec-Fetch-Site`` http headers to bypass tusksearch's latest bot detection.
2026-09-19 07:30:06 +02:00
Bnyro
c0ec29fdc3 [fix] pixabay: crashes when there are no results 2026-09-18 17:18:59 +02:00
Markus Heiser
280aceb1ae [mod] add new category 'stock_images' - a specialization of 'images' (#6703)
Engines like Pexels provide images that one would more likely expect in a
category named ``stock_images``.

By classifying engines like Pexels more precisely, we enable a more specific
handling of them, which also helps to avoid mixing them with the more general
image search on the internet. [1]

We leave these engines *enabled* by default, take them out of ``images`` and
move them into the specialization ``stock_images``.

This makes it possible, on the one hand, for the admin to configure a tab in the
UI, and on the other hand, the user can always select the group directly using
the search syntax ``!stock_images ...``.

What we need to keep in mind:

1. Technically speaking, there are no subcategories in the strict
   sense.  Organizationally, the *subcategories* are derived from the
   `categories_as_tabs`.

2. From an admin’s perspective, `categories_as_tabs` is the tool for
   structuring their content.

3. In the context of “enabled/disabled by default”: We should avoid having
   the admin have to activate many individual engines when they want to
   structure their content (UI tabs).

So if we leave an engine enabled by default and **move** it from `images` to a
new category, the admin only needs to add the new category to
`categories_as_tabs` to restructure their content.  If the new category is also
part of the translations (see `searx/searxng.msg` in this PR), this UI structure
would also be available internationally.

[1] https://github.com/searxng/searxng/pull/6690#issuecomment-5646291679

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2026-09-18 16:13:42 +02:00
Bnyro
1f7711ef1f [mod] categories: move app search engines to it category instead of files 2026-09-18 13:19:28 +02:00
Bnyro
12df7aed60 [del] library of congress: now behind cloudflare (#6756) 2026-09-18 12:27:30 +02:00
Bnyro
96c35f1ed3 [fix] findfiles images: crashes due to changed HTML layout 2026-09-18 11:47:45 +02:00
Bnyro
087ec6fbda [fix] 500px: crashes when there's no description (#6745) 2026-09-18 07:42:08 +02:00
Bnyro
fc028154a4 [fix] giphy: API key extraction fails (#6744) 2026-09-18 07:41:40 +02:00
Bnyro
274b63b677 [refactor] engines: migrate to SXNG_Response.html() (#6718) 2026-09-17 12:22:38 +02:00
Bnyro
c497719921 [feat] yandex web: add time range support 2026-09-17 09:46:21 +02:00
Bnyro
e57ebb0aa5 [refactor] engine processors: add type for time range and safesearch in OnlineParams 2026-09-17 09:46:21 +02:00
Bnyro
77d07c8743 [mod] yandex: simplify request method and drop unused params/cookies
We previously set the 'cookie' cookie value with a timestamp
from 2024 to `yq=...`. The cookie is not actually used in Yandex,
instead we probably wanted to set the `yq` cookie instead. But
since the engine works without it, it can be dropped.

Also simplifies the request method a bit to be easier to read.
2026-09-17 09:46:21 +02:00
Bnyro
3accbaca79 [feat] mojeek: automatically solve altcha captchas 2026-09-17 09:00:33 +02:00
Markus Heiser
461f174b09 [mod] engine: pexels - added type hints and migrate to Image-Results (#6739)
Code style somewhat unified, type annotations added, ``LegacyResult`` replaced
by ``Image``, but no functional modifications.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2026-09-16 16:46:16 +02:00
Markus Heiser
1354f3952b [feat] mediathekviewweb: add description to queried fields (#6717)
Queries like `!mvw hercule poirot` usually yield no results if the `description`
is not also included in the search.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2026-09-16 16:35:47 +02:00
Bnyro
f9c9c35546 [fix] mediathekviewweb: crashes when result has no hd video url (#6717) 2026-09-16 16:35:47 +02:00
Bnyro
71a2424c24 [feat] mediathekviewweb: modernize engine to use EngineResults (#6717) 2026-09-16 16:35:47 +02:00
WookieeOnCrack
f725cc7936 [fix] yandex: image result xpath + modernization (#6728) 2026-09-16 10:38:32 +02:00
vojkovic
ca49650407 [fix] engines: yahoo modernisation and ybv cookie 2026-09-15 17:37:19 +08:00
vojkovic
604ee1698c [fix] engines: flaticon encode non ascii words 2026-09-15 08:50:46 +08:00
Ivan Gabaldon
d4ce87c234 [mod] py: format (#6560) 2026-09-13 23:48:01 +02:00
Bnyro
3b80090654 [del] adobe stock: remove because it's behind a datadome captcha 2026-09-13 14:49:03 +02:00
Bnyro
e61d097562 [fix] 500px: migrate to new search endpoint 2026-09-13 14:48:44 +02:00
Bnyro
32f2da4ef0 [fix] public domain image archive: fails to extract API url 2026-09-13 13:43:01 +02:00
Markus Heiser
a667e8c905 [mod] openverse - use of internal APIs and addition result fields (#6701)
Additional fields:
  add resolution, thumbnail and author to the image result

Types:
  use EngineResults and add type hints from internall APIs

Disabled engine:
  as of 09/2026: The availability of https://openverse.org/ is poor, and queries
  and image requests often time out. However, these issues also occur when using
  a WEB browser to search on openverse.org or view images.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2026-09-12 18:18:32 +02:00
vojkovic
0a94da4368 [fix] engines: naver images updated matcher 2026-09-12 18:17:41 +02:00
Markus Heiser
56b1f64541 [mod] engine: bing images - use internal API & type hints (#6699)
Switching to internal APIs such as ``EngineResults`` and
adding more type hints.  No functional changes.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2026-09-12 11:22:01 +02:00
Markus Heiser
923a307454 [fix] engine: bing images - remove market place argument mkt (#6699)
The ``mkt`` argument does no longer exists and the ``async`` argument has been
renamed to ``mmasync``.

For still unknown reasons (IP based?), some users had to observe that the title
was missing [1].  I myself was not able to reproduce this error, however the
evaluation of the title was additionally expanded by the attribute value of the
``<a title=".."`` element.

Related:

- [1] https://github.com/searxng/searxng/pull/6690#issuecomment-5636811572

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2026-09-12 11:22:01 +02:00
vojkovic
87bf8c86ed [fix] engines: dogpile web needs api token 2026-09-12 16:32:42 +08:00
Markus Heiser
6a27c21008 [fix] engine: pinterest - fix empty titles and complete modernization (#6694)
* [fix] engine: pinterest - fix empty titles and complete modernization

The titles from the fields ``title`` and ``grid_title`` are mostly empty or have
short strings without meaningful content.  Various fields for the title are now
being queried, which have more informative value.

As part of the bug fix, the engine was completely revised and modernized.

Related:

- https://github.com/searxng/searxng/pull/6690#issuecomment-5631113121

Co-authored-by: @vojkovic
2026-09-11 16:24:58 +02:00
Brock Vojkovic
ffe96f8a6f [fix] engines: bing first word results (#6671)
Fixes the bing web engine, it was just using the first word of the query for the search and return random junk other times. see: vojkovic#10

Swapped to use bing's setlang and cc params. I found us, cn, ru return complete garbage 100% of the time. I reckon that if you don't have an ip address from there it will just return garbage, so those three are skipped. Also removed accept language override because it didn't change anything anymore.


- Closes: https://github.com/searxng/searxng/issues/4964
- Related: https://github.com/vojkovic/searxng/issues/10
2026-09-11 08:41:13 +02:00
Markus Heiser
931fd9787b [fix] engine: core.ac.uk - don't split a string into a tag list (#6689) 2026-09-10 14:45:43 +02:00
vojkovic
42e1d61296 [fix] engines: startpage anubis solver 2026-09-10 19:12:21 +08:00
vojkovic
765a9999df [fix] engines: duckduckgo web bypass botdetection 2026-09-10 18:56:12 +08:00
rdurnik
ba055b3e09 [feat] engines: add europepmc (science, scientific publications) 2026-09-10 11:27:28 +02:00
Markus Heiser
3fdc6d753a [del] remove searx_engine - no use case (#6680)
There is no meaningful use-case for the engine, moreover the implementation
is 10 years old (formerly searx) and no longer fits the current data model of
SearXNG.

Closes: - https://github.com/searxng/searxng/issues/6676

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2026-09-08 09:29:41 +02:00
Ted Lanham
3e454637fb [fix] braveapi: set JSON Accept header (#6666) 2026-09-07 13:18:07 +02:00
Bnyro
c06e9f0889 [del] cara engine: remove because it requires signup for searching 2026-09-05 22:44:32 +02:00
Bnyro
4781754dc4 [mod] engines: remove uses of enable_http2 = False
This has only been needed because we were flagged by HTTP2 fingerprinting.
Now, since we use curl cffi, we can bypass the fingerpinting, so we can
use HTTP2 just fine without getting blocked.
2026-09-05 19:51:58 +02:00
Bnyro
28b61729c7 [fix] resulthunter: detect blocked requests instead of returning 0 results 2026-09-05 19:47:23 +02:00
Bnyro
14a9f84c6c [fix] chatnoir: returns empty responses 2026-09-05 19:46:48 +02:00
vojkovic
8b01679e8f [fix] engines: update brave images/videos parser and news xpath 2026-09-06 01:45:40 +08:00
vojkovic
eaf1fcb349 [fix] engines: resulthunter search_source arg 2026-09-06 01:43:11 +08:00
vojkovic
e20e370353 [fix] engines: tusksearch referer header 2026-09-06 01:38:09 +08:00
vojkovic
3605a2d58b [fix] engines: dogpile request api token 2026-09-06 01:22:09 +08:00
Bnyro
aef258321c [feat] engines: add searchrockit.com (general, news, images) 2026-09-05 19:16:29 +02:00
Bnyro
a303e9c0ca [fix] neosearch: requests blocked due to missing xsrf token 2026-09-05 19:14:16 +02:00
Bnyro
ccffbfc164 [fix] searchzee: bypass botblocking
requires https://github.com/searxng/searxng/pull/6620
2026-09-05 19:11:49 +02:00
Markus Heiser
15a91992e4 [docs] fix some minor reST markup issues in the doc-string (#6646)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2026-09-04 11:18:59 +02:00