jandieman/searxng
2
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2026-09-23 06:36:12 +00:00
Code Issues Projects Releases Wiki Activity
9,773 Commits 5 Branches 0 Tags
fe5bdd1bf2c4d6223cb56cd5d3f07106a0a408fe
Commit Graph

9773 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Bnyro
fe5bdd1bf2 [feat] magnific: solve captcha to bypass botblocking 
Magnific now employs a very simple to solve CAPTCHA before
all API requests (essentially summing up some numbers).

Python script for testing:
```python
import re
from lxml import html
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:156.0) Gecko/20100101 Firefox/156.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.9',
    'Sec-GPC': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'same-origin',
    'Priority': 'u=0, i',
    'Pragma': 'no-cache',
    'Cache-Control': 'no-cache',
}

resp = requests.get(
    "https://www.magnific.com/api/regular/search?term=tree&filters%5Bai-generated%5D%5Bexcluded%5D=1&page=1&locale=en&filters%5Blicense%5D=free",
    headers=headers,
)
cookies = resp.cookies
print(resp.text)

_NUMBER_RE = re.compile("\d+")
_BM_VERIFY = re.compile(r"\"bm-verify\":\s*\"(.*?)\"")

doc = html.fromstring(resp.text)
script = doc.xpath("//script")[0].text
numbers = [int(m) for m in _NUMBER_RE.findall(script)]
bm_verify = _BM_VERIFY.search(resp.text)
if not bm_verify:
    exit(1)
bm_verify = bm_verify.group(1)
print("bm-verify: " + bm_verify)

solution = sum(numbers)
print(f"sum({numbers}) = {solution}")

resp = requests.post(
    "https://www.magnific.com/_sec/verify?provider=interstitial",
    json={"bm-verify": bm_verify, "pow": solution},
    headers=headers,
    cookies=cookies,
)

resp = requests.get(
    "https://www.magnific.com/api/regular/search?term=bird&filters%5Bai-generated%5D%5Bexcluded%5D=1&page=1&locale=en&filters%5Blicense%5D=free",
    headers=headers,
    cookies=resp.cookies,
)
print(resp.text)
```
 2026-09-19 11:31:00 +02:00
Brock Vojkovic
367fb6537c [fix] engines: tusksearch cors request headers (#6758) 
Added the ``Origin`` and ``Sec-Fetch-Site`` http headers to bypass tusksearch's latest bot detection.
 2026-09-19 07:30:06 +02:00
dependabot[bot]
c0042add30 [upd] web-client (simple): Bump the minor group in /client/simple with 3 updates (#6750) 
* [upd] web-client (simple): Bump the minor group

Bumps the minor group in /client/simple with 3 updates: [@biomejs/biome](https://github.com/biomejs/biome/tree/HEAD/packages/@biomejs/biome), [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) and [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite).

Updates `@biomejs/biome` from 2.5.12 to 2.5.13
- [Release notes](https://github.com/biomejs/biome/releases)
- [Changelog](https://github.com/biomejs/biome/blob/main/packages/@biomejs/biome/CHANGELOG.md)
- [Commits](https://github.com/biomejs/biome/commits/@biomejs/biome@2.5.13/packages/@biomejs/biome)

Updates `@types/node` from 26.5.0 to 26.5.1
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `vite` from 8.2.2 to 8.3.0
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/create-vite@8.3.0/packages/vite)

---
updated-dependencies:
- dependency-name: "@biomejs/biome"
  dependency-version: 2.5.13
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor
- dependency-name: "@types/node"
  dependency-version: 26.5.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor
- dependency-name: vite
  dependency-version: 8.3.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* [upd] sync lockfile

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ivan Gabaldon <igabaldon@inetol.net>
 2026-09-18 17:23:06 +02:00
Bnyro
c0ec29fdc3 [fix] pixabay: crashes when there are no results 2026-09-18 17:18:59 +02:00
dependabot[bot]
579809a053 [upd] github-actions: Bump docker/setup-qemu-action from 4.3.0 to 4.4.0 (#6747) 
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.3.0 to 4.4.0.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](1f40c72289...9901266195)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: 4.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-09-18 17:10:17 +02:00
dependabot[bot]
9cde918dd2 [upd] github-actions: Bump actions/download-artifact from 4.3.0 to 8.0.1 (#6748) 
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 4.3.0 to 8.0.1.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](d3f86a106a...3e5f45b2cf)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-09-18 17:08:07 +02:00
dependabot[bot]
63b827edc7 [upd] github-actions: Bump actions/upload-artifact from 4.6.2 to 7.0.1 (#6749) 
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.6.2 to 7.0.1.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](ea165f8d65...043fb46d1a)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-09-18 17:07:54 +02:00
Bnyro
74ffd01152 [feat] engines: add littlelayer (general / indie) engine 2026-09-18 16:45:36 +02:00
Markus Heiser
280aceb1ae [mod] add new category 'stock_images' - a specialization of 'images' (#6703) 
Engines like Pexels provide images that one would more likely expect in a
category named ``stock_images``.

By classifying engines like Pexels more precisely, we enable a more specific
handling of them, which also helps to avoid mixing them with the more general
image search on the internet. [1]

We leave these engines *enabled* by default, take them out of ``images`` and
move them into the specialization ``stock_images``.

This makes it possible, on the one hand, for the admin to configure a tab in the
UI, and on the other hand, the user can always select the group directly using
the search syntax ``!stock_images ...``.

What we need to keep in mind:

1. Technically speaking, there are no subcategories in the strict
   sense.  Organizationally, the *subcategories* are derived from the
   `categories_as_tabs`.

2. From an admin’s perspective, `categories_as_tabs` is the tool for
   structuring their content.

3. In the context of “enabled/disabled by default”: We should avoid having
   the admin have to activate many individual engines when they want to
   structure their content (UI tabs).

So if we leave an engine enabled by default and **move** it from `images` to a
new category, the admin only needs to add the new category to
`categories_as_tabs` to restructure their content.  If the new category is also
part of the translations (see `searx/searxng.msg` in this PR), this UI structure
would also be available internationally.

[1] https://github.com/searxng/searxng/pull/6690#issuecomment-5646291679

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2026-09-18 16:13:42 +02:00
Bnyro
1f7711ef1f [mod] categories: move app search engines to it category instead of files 2026-09-18 13:19:28 +02:00
Bnyro
12df7aed60 [del] library of congress: now behind cloudflare (#6756) 2026-09-18 12:27:30 +02:00
Bnyro
96c35f1ed3 [fix] findfiles images: crashes due to changed HTML layout 2026-09-18 11:47:45 +02:00
dependabot[bot]
0d6910ae5a [upd] pypi: Bump the minor group with 3 updates (#6751) 
Bumps the minor group with 3 updates: [granian](https://github.com/emmett-framework/granian), [selenium](https://github.com/SeleniumHQ/Selenium) and [basedpyright](https://github.com/detachhead/basedpyright).


Updates `granian` from 2.8.2 to 2.8.3
- [Release notes](https://github.com/emmett-framework/granian/releases)
- [Commits](https://github.com/emmett-framework/granian/compare/v2.8.2...v2.8.3)

Updates `selenium` from 4.48.0 to 4.49.0
- [Release notes](https://github.com/SeleniumHQ/Selenium/releases)
- [Commits](https://github.com/SeleniumHQ/Selenium/compare/selenium-4.48.0...selenium-4.49.0)

Updates `basedpyright` from 1.40.0 to 1.40.1
- [Release notes](https://github.com/detachhead/basedpyright/releases)
- [Commits](https://github.com/detachhead/basedpyright/compare/v1.40.0...v1.40.1)
 2026-09-18 09:32:40 +02:00
Bnyro
087ec6fbda [fix] 500px: crashes when there's no description (#6745) 2026-09-18 07:42:08 +02:00
Bnyro
fc028154a4 [fix] giphy: API key extraction fails (#6744) 2026-09-18 07:41:40 +02:00
Bnyro
274b63b677 [refactor] engines: migrate to SXNG_Response.html() (#6718) 2026-09-17 12:22:38 +02:00
Bnyro
3169aafd28 [feat] SXNG_Response: add method to parse html body (#6718) 2026-09-17 12:22:38 +02:00
Bnyro
c497719921 [feat] yandex web: add time range support 2026-09-17 09:46:21 +02:00
Bnyro
e57ebb0aa5 [refactor] engine processors: add type for time range and safesearch in OnlineParams 2026-09-17 09:46:21 +02:00
Bnyro
77d07c8743 [mod] yandex: simplify request method and drop unused params/cookies 
We previously set the 'cookie' cookie value with a timestamp
from 2024 to `yq=...`. The cookie is not actually used in Yandex,
instead we probably wanted to set the `yq` cookie instead. But
since the engine works without it, it can be dropped.

Also simplifies the request method a bit to be easier to read.
 2026-09-17 09:46:21 +02:00
Brock Vojkovic
9aab16aef7 [fix] network: stop http redirects (#6740) 
Stop curl from following from a HTTPS origin to a HTTP page

See: https://curl.se/libcurl/c/CURLOPT_REDIR_PROTOCOLS_STR.html

Reported-by: Rohit Dixit <00.00.xit@gmail.com>
 2026-09-17 15:27:01 +08:00
Bnyro
3accbaca79 [feat] mojeek: automatically solve altcha captchas 2026-09-17 09:00:33 +02:00
Bnyro
9d82be9d89 [feat] utils.py: add altcha captcha solver 2026-09-17 09:00:33 +02:00
Markus Heiser
461f174b09 [mod] engine: pexels - added type hints and migrate to Image-Results (#6739) 
Code style somewhat unified, type annotations added, ``LegacyResult`` replaced
by ``Image``, but no functional modifications.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2026-09-16 16:46:16 +02:00
Markus Heiser
1354f3952b [feat] mediathekviewweb: add description to queried fields (#6717) 
Queries like `!mvw hercule poirot` usually yield no results if the `description`
is not also included in the search.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2026-09-16 16:35:47 +02:00
Bnyro
f9c9c35546 [fix] mediathekviewweb: crashes when result has no hd video url (#6717) 2026-09-16 16:35:47 +02:00
Bnyro
71a2424c24 [feat] mediathekviewweb: modernize engine to use EngineResults (#6717) 2026-09-16 16:35:47 +02:00
Brock Vojkovic
ce19e9fae3 [fix] image_proxy compressed images (#6730) 
Prevents the image proxy from decompressing brotli/gzip in transit.

See: https://curl.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html

Setting accept_encoding to none prevents curl from automatically decompressing the received contents.

Signed-off-by: vojkovic <git@vojk.au>
 2026-09-16 22:23:11 +08:00
WookieeOnCrack
f725cc7936 [fix] yandex: image result xpath + modernization (#6728) 2026-09-16 10:38:32 +02:00
vojkovic
ca49650407 [fix] engines: yahoo modernisation and ybv cookie 2026-09-15 17:37:19 +08:00
vojkovic
94218a3ac6 [build] /static 2026-09-15 09:56:24 +08:00
vojkovic
beb324c0f5 [mod] simple: apply nojs via stylesheet 2026-09-15 09:56:24 +08:00
vojkovic
604ee1698c [fix] engines: flaticon encode non ascii words 2026-09-15 08:50:46 +08:00
Ivan Gabaldon
ef05645f09 [mod] container: rework builds (#6720) 
Free from ghcr.io/searxng/cache! Now we save and load images without polluting
this shared (and useless) layer cache, which also makes it easy to download
artifacts for local debugging.

I also removed this nonsense about virtualenv normalization, this will be
properly fixed by using lockfiles on py dependencies.
 2026-09-14 16:06:00 +02:00
Ivan Gabaldon
d4ce87c234 [mod] py: format (#6560) 2026-09-13 23:48:01 +02:00
dependabot[bot]
ab5dd7599e [upd] pypi: Bump black from 25.9.0 to 26.5.1 (#6560) 
Bumps [black](https://github.com/psf/black) from 25.9.0 to 26.5.1.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/25.9.0...26.5.1)

---
updated-dependencies:
- dependency-name: black
  dependency-version: 26.5.1
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2026-09-13 23:48:01 +02:00
vojkovic
230c3632d9 [mod] github: convert issue templates to yaml forms 2026-09-13 22:37:48 +08:00
Bnyro
42829cecd6 [feat] results: limit titles and content to fixed length 2026-09-13 15:05:47 +02:00
Bnyro
3b80090654 [del] adobe stock: remove because it's behind a datadome captcha 2026-09-13 14:49:03 +02:00
Bnyro
e61d097562 [fix] 500px: migrate to new search endpoint 2026-09-13 14:48:44 +02:00
Bnyro
32f2da4ef0 [fix] public domain image archive: fails to extract API url 2026-09-13 13:43:01 +02:00
Brock Vojkovic
96479fff65 [fix] ci: run armv7 test on the amd64 runner (#6711) 2026-09-13 09:25:21 +02:00
dependabot[bot]
d4f00d15d4 [upd] pypi: Bump the minor group with 4 updates (#6693) 
Bumps the minor group with 4 updates: [lxml](https://github.com/lxml/lxml), [curl-cffi](https://github.com/lexiforest/curl_cffi), [wlc](https://github.com/WeblateOrg/wlc) and [basedpyright](https://github.com/detachhead/basedpyright).


Updates `lxml` from 6.1.2 to 6.1.3
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-6.1.2...lxml-6.1.3)

Updates `curl-cffi` from 0.16.1 to 0.16.3
- [Release notes](https://github.com/lexiforest/curl_cffi/releases)
- [Changelog](https://github.com/lexiforest/curl_cffi/blob/main/docs/changelog.rst)
- [Commits](https://github.com/lexiforest/curl_cffi/compare/v0.16.1...v0.16.3)

Updates `wlc` from 2.1.1 to 2.2.0
- [Release notes](https://github.com/WeblateOrg/wlc/releases)
- [Changelog](https://github.com/WeblateOrg/wlc/blob/main/CHANGES.md)
- [Commits](https://github.com/WeblateOrg/wlc/compare/2.1.1...2.2.0)

Updates `basedpyright` from 1.39.10 to 1.40.0
- [Release notes](https://github.com/detachhead/basedpyright/releases)
- [Commits](https://github.com/detachhead/basedpyright/compare/v1.39.10...v1.40.0)
 2026-09-12 18:20:51 +02:00
Markus Heiser
a667e8c905 [mod] openverse - use of internal APIs and addition result fields (#6701) 
Additional fields:
  add resolution, thumbnail and author to the image result

Types:
  use EngineResults and add type hints from internall APIs

Disabled engine:
  as of 09/2026: The availability of https://openverse.org/ is poor, and queries
  and image requests often time out. However, these issues also occur when using
  a WEB browser to search on openverse.org or view images.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2026-09-12 18:18:32 +02:00
vojkovic
0a94da4368 [fix] engines: naver images updated matcher 2026-09-12 18:17:41 +02:00
Markus Heiser
56b1f64541 [mod] engine: bing images - use internal API & type hints (#6699) 
Switching to internal APIs such as ``EngineResults`` and
adding more type hints.  No functional changes.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2026-09-12 11:22:01 +02:00
Markus Heiser
923a307454 [fix] engine: bing images - remove market place argument mkt (#6699) 
The ``mkt`` argument does no longer exists and the ``async`` argument has been
renamed to ``mmasync``.

For still unknown reasons (IP based?), some users had to observe that the title
was missing [1].  I myself was not able to reproduce this error, however the
evaluation of the title was additionally expanded by the attribute value of the
``<a title=".."`` element.

Related:

- [1] https://github.com/searxng/searxng/pull/6690#issuecomment-5636811572

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2026-09-12 11:22:01 +02:00
vojkovic
87bf8c86ed [fix] engines: dogpile web needs api token 2026-09-12 16:32:42 +08:00
dependabot[bot]
61d660276f [upd] web-client (simple): Bump the minor group (#6692) 
Bumps the minor group in /client/simple with 2 updates: [@biomejs/biome](https://github.com/biomejs/biome/tree/HEAD/packages/@biomejs/biome) and [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node).


Updates `@biomejs/biome` from 2.5.11 to 2.5.12
- [Release notes](https://github.com/biomejs/biome/releases)
- [Changelog](https://github.com/biomejs/biome/blob/main/packages/@biomejs/biome/CHANGELOG.md)
- [Commits](https://github.com/biomejs/biome/commits/@biomejs/biome@2.5.12/packages/@biomejs/biome)

Updates `@types/node` from 26.4.1 to 26.5.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@biomejs/biome"
  dependency-version: 2.5.12
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: minor
- dependency-name: "@types/node"
  dependency-version: 26.5.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-09-11 16:57:14 +02:00
Markus Heiser
6a27c21008 [fix] engine: pinterest - fix empty titles and complete modernization (#6694) 
* [fix] engine: pinterest - fix empty titles and complete modernization

The titles from the fields ``title`` and ``grid_title`` are mostly empty or have
short strings without meaningful content.  Various fields for the title are now
being queried, which have more informative value.

As part of the bug fix, the engine was completely revised and modernized.

Related:

- https://github.com/searxng/searxng/pull/6690#issuecomment-5631113121

Co-authored-by: @vojkovic
 2026-09-11 16:24:58 +02:00