Commit Graph

6675 Commits

Author SHA1 Message Date
Alexandre Flament e81c458165
Merge pull request #1656 from searxng/dependabot/pip/master/flask-2.2.2
Bump flask from 2.1.3 to 2.2.2
2022-08-13 18:33:39 +02:00
Markus Heiser 0043190643
Merge pull request #1655 from return42/fix-opensearch
[fix] improve OpenSearch description
2022-08-13 12:36:34 +02:00
Léon Tiekötter b56848e8d0
Merge pull request #1660 from liimee/eng-neeva
Add neeva engine
2022-08-13 02:53:27 +02:00
Léon Tiekötter 8fa84ee9f4 Fix time range support and add soft_max_redirects
Add custom time_range_url and time_range_map

Set soft_max_redirects = 2 to prevent "ErrorContext('searx/search/processors/online.py', 116, 'count_error(', None, '2 redirects, maximum: 0', ('200', 'OK', 'neeva.com')) True"
2022-08-13 07:35:15 +07:00
ta d6dfada1a9 add neeva web engine
Neeva is "the world's first ad-free, private search engine" and uses data from Apple, Bing, Yelp and "others".
They claim to crawl "hundreds of millions" of URLs a day (https://twitter.com/Neeva/status/1536447373903335426).
2022-08-13 07:26:36 +07:00
Markus Heiser 561d24cd2c
Merge pull request #1657 from searxng/dependabot/pip/master/selenium-4.4.0
Bump selenium from 4.3.0 to 4.4.0
2022-08-12 18:54:50 +02:00
Markus Heiser 334eccfad6
Merge pull request #1658 from searxng/dependabot/pip/master/setproctitle-1.3.2
Bump setproctitle from 1.3.1 to 1.3.2
2022-08-12 18:54:19 +02:00
Markus Heiser 6481beb479
Merge pull request #1659 from searxng/translations_update
Update translations
2022-08-12 18:51:08 +02:00
searxng-bot b4fea007f5 [translations] update from Weblate
f2997bfa - 2022-08-12 - Markus Heiser <markus.heiser@darmarit.de>
eeca674f - 2022-08-10 - Edrean Ernst <edrean@allesbeste.com>
7478de6a - 2022-08-11 - Markus Heiser <markus.heiser@darmarit.de>
c4fb9110 - 2022-08-07 - wordpure <wordlesspure@gmail.com>
a5b432e2 - 2022-08-11 - Markus Heiser <markus.heiser@darmarit.de>
eb01d415 - 2022-08-09 - Markus Heiser <markus.heiser@darmarit.de>
f96eb06e - 2022-08-11 - Shopimisrel <shopisrael12@gmail.com>
e7c79191 - 2022-08-08 - ajnasaboobacker <ajnasaboobacker@gmail.com>
f4dbd424 - 2022-08-08 - ajnasaboobacker <ajnasaboobacker@gmail.com>
2022-08-12 07:18:14 +00:00
dependabot[bot] bf36057c79
Bump setproctitle from 1.3.1 to 1.3.2
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/commits)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-12 07:04:32 +00:00
dependabot[bot] 11dfff766b
Bump selenium from 4.3.0 to 4.4.0
Bumps [selenium](https://github.com/SeleniumHQ/Selenium) from 4.3.0 to 4.4.0.
- [Release notes](https://github.com/SeleniumHQ/Selenium/releases)
- [Commits](https://github.com/SeleniumHQ/Selenium/compare/selenium-4.3.0...selenium-4.4.0)

---
updated-dependencies:
- dependency-name: selenium
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-12 07:04:30 +00:00
dependabot[bot] 828073357e
Bump flask from 2.1.3 to 2.2.2
Bumps [flask](https://github.com/pallets/flask) from 2.1.3 to 2.2.2.
- [Release notes](https://github.com/pallets/flask/releases)
- [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/flask/compare/2.1.3...2.2.2)

---
updated-dependencies:
- dependency-name: flask
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-12 07:04:25 +00:00
Markus Heiser 3b0f9c07b2 [fix] improve OpenSearch description
Some HTTP-Clients do have issues with the ``opensearch.xml`` from SearXNG
(related [1][2]) while other OpenSearch descriptions[3] (e.g. from qwant) work
flawles.

Inspired by the OpenSearch description from qwant and with informations from the
specification[4] the ``opensearch.xml`` has been *improved*.

- convert `<Url>` methods from lower case to upper case (`POST`|`GET`)
- add `<moz:SearchForm>` and `xmlns:moz="http://www.mozilla.org/2006/browser/search/"`
- add `<Query role="example" searchTerms="SearXNG" />`  [4]

  OpenSearch description documents should include at least one Query element of
  `role="example"` that is expected to return search results. Search clients may
  use this example query to validate that the search engine is working properly.

- modified `<LongName>` to SearXNG
- modified `<Description>` the word 'hackable' scares uninitiated users and was removed
- add the `type="image/png"` to `<Image>`

Test can be done by::

    make run

Visit http://127.0.0.1:8888/ and add the search engine to your WEB-Browser /
test with different WEB-Browser from desktop and Smartphones (are there any iOS
user here, please test on Safari and Chrome).

[1] https://app.element.io/#/room/#searxng:matrix.org/$xN_abdKhNqUlgXRBrb_9F3pqOxnSzGQ1TG0s0G9hQVw
[2] https://github.com/searxng/searxng/issues/431
[3] https://developer.mozilla.org/en-US/docs/Web/OpenSearch
[4] https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md#the-query-element

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-11 19:04:36 +02:00
Markus Heiser 2bfb269f0a
Merge pull request #1618 from return42/drop-pycodestyle
[fix] pycodestyle has been replaced by black in 3c77412d3
2022-08-10 21:58:16 +02:00
Markus Heiser 214f234c6f
Merge pull request #1649 from joxcat/eng-srht
Add sourcehut engine
2022-08-10 21:56:25 +02:00
Markus Heiser 73ffa0c070 Update searx.data - update_engine_descriptions.py
./manage pyenv.cmd python ./searxng_extra/update/update_engine_descriptions.py

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-10 21:48:34 +02:00
Johan Planchon e0215e0cc8 add sourcehut engine 2022-08-10 21:48:12 +02:00
Markus Heiser 43c545910b
Merge pull request #1648 from unixfox/google_mobile_ui_parameter
output format protobuf to HTML for google mobile
2022-08-10 20:50:45 +02:00
Markus Heiser eb02cc77c5 [fix] google - simplify XPath selectors to fetch more results
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-10 18:55:31 +02:00
Markus Heiser 5706d110f2
Merge pull request #1650 from joxcat/eng-librs
Add lib.rs engine
2022-08-10 17:45:28 +02:00
Johan Planchon 90bdb23756 add lib.rs engine 2022-08-10 12:17:55 +02:00
Émilien Devos b9f16a77db output format protobuf to HTML for google mobile 2022-08-10 09:36:06 +00:00
Thomas Renard d4acbcfe63 [mod] add deepl translation engine
This implements the Deepl Translation engine. It works nearly like lingva but
directly to the deepl API.  This api only needs a to-lang, from-lang is a fake
by now.

There is a free option to use [1].

[1] https://www.deepl.com/pro-api?cta=header-pro-api for registering a free account.
2022-08-10 09:14:36 +02:00
Léon Tiekötter 07342899e5
Merge pull request #1643 from vojkovic/master
Revert Google XPath.
2022-08-10 04:18:42 +02:00
Brock Vojković 24210fb10b
Revert PR #1633
This reverts the changes made to the Google results XPath in PR #1633.
2022-08-10 03:41:39 +02:00
Markus Heiser 3d6b098a5f
Merge pull request #1633 from tiekoetter/fix-google-results-xpath
[fix] google engine: results XPath
2022-08-09 06:28:03 +02:00
Léon Tiekötter 94b3656b4a [fix] google engine: results XPath
Seems google rolls out changes first on the `google.com` domain and later on the
"language" domains.  By example: yesterday [1] `google.com` did not work but
`google.de` and `google.fr` did work, today they do not work any longer and this
fix is needed on all domains.

Closes: https://github.com/searxng/searxng/issues/1628
[1] https://github.com/searxng/searxng/issues/1628#issuecomment-1208191816
2022-08-09 06:23:59 +02:00
Markus Heiser 925f3ff487
Merge pull request #1627 from allendema/add-marginalia
[enh] Initial Marginalia.nu support (foss)
2022-08-08 06:53:02 +02:00
Markus Heiser 58f630d312
Merge pull request #1623 from return42/mod-yep.com
[mod] engine yep.com: show all 100 results yep.com has
2022-08-08 06:52:15 +02:00
Allen 5b815b3966 [enh] initial Marginalia.nu support (foss)
Currently it uses a public api_key `/public/` [1]

The 'index' parameter selects the search index, corresponding to the drop down
next to the search field in the main GUI.

    0: popular
    1: blogs
    2: big_sites
    3: default
    4: experimental

'experimental' is more up to date and does not exclude other sites, which is the
case with 'big sites' or 'blogs'.

[1] https://api.marginalia.nu/
[2] https://git.marginalia.nu/marginalia/marginalia.nu
[3] https://news.ycombinator.com/item?id=31536626

Closes: https://github.com/searxng/searxng/issues/1620
2022-08-08 06:31:04 +02:00
Markus Heiser 04e3427022
Merge pull request #1626 from return42/fix-minor
[fix] typo in uWSGI cache: searxcache --> searxngcache
2022-08-07 12:53:33 +02:00
Markus Heiser 8b58cd1167 [fix] typo in uWSGI cache: searxcache --> searxngcache
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-07 12:31:09 +02:00
liimee 8c318562e2
add description and wikidata ID to wttr.in engine 2022-08-07 14:57:10 +07:00
ta 8aa018db95 add wttr.in engine 2022-08-07 13:04:18 +07:00
Markus Heiser 647e6187d7 [mod] engine yep.com: show all 100 results yep.com has
yep.com is still in beta, the api.yep.com does not have paging support.  There
is only a 'limit' argument with a maximum of 100 results.

yep.com seems fast; there is nor need for a timeout of 12 sec.

The API returns JSON nevertheless what the HTTP header is, the "show more"
button on yep.com's web site does not set a special HTTP Accept header.

FYI: The index does not support languages, the WEB UI does not offer a language
selection of the results and the entire index seems in English.

Closes: https://github.com/searxng/searxng/issues/1619
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-06 13:21:28 +02:00
Markus Heiser 63f5121b38 [fix] pycodestyle has been replaced by black in 3c77412d3
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-05 10:29:58 +02:00
Markus Heiser cee586029c
Merge pull request #1614 from searxng/dependabot/pip/master/setproctitle-1.3.1
Bump setproctitle from 1.3.0 to 1.3.1
2022-08-05 10:03:35 +02:00
Markus Heiser 3d0bf840b9
Merge pull request #1617 from searxng/translations_update
Update translations
2022-08-05 09:44:57 +02:00
Markus Heiser 051cf5bc61
Merge pull request #1615 from searxng/dependabot/pip/master/pycodestyle-2.9.1
Bump pycodestyle from 2.8.0 to 2.9.1
2022-08-05 09:33:05 +02:00
searxng-bot 0f02fcc836 [translations] update from Weblate
acf8bd39 - 2022-08-05 - Markus Heiser <markus.heiser@darmarit.de>
4ad75b6e - 2022-08-04 - Markus Heiser <markus.heiser@darmarit.de>
ee8cbee6 - 2022-07-31 - Markus Heiser <markus.heiser@darmarit.de>
87c19313 - 2022-08-01 - Academic tyro <y13593582403@gmail.com>
cbe0de32 - 2022-07-30 - Markus Heiser <markus.heiser@darmarit.de>
45029a17 - 2022-08-04 - Markus Heiser <markus.heiser@darmarit.de>
6eec3795 - 2022-08-03 - Markus Heiser <markus.heiser@darmarit.de>
f8d8f31f - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
b3fb365f - 2022-07-29 - Markus Heiser <markus.heiser@darmarit.de>
aaeabbc9 - 2022-08-03 - Lakatos Tamás <tomimost@gmail.com>
6c71c501 - 2022-08-03 - Markus Heiser <markus.heiser@darmarit.de>
f7b5ba19 - 2022-08-01 - Markus Heiser <markus.heiser@darmarit.de>
850e7fa0 - 2022-08-04 - Mico Hautaluoma <m@mha.fi>
0cb696fc - 2022-07-31 - Markus Heiser <markus.heiser@darmarit.de>
04c3785f - 2022-08-02 - Markus Heiser <markus.heiser@darmarit.de>
b500f2ad - 2022-08-01 - Edrean Ernst <edrean@allesbeste.com>
0b576b83 - 2022-08-01 - GooGuJiang <gu@gmoe.cc>
0adeb6e2 - 2022-08-01 - Edrean Ernst <edrean@allesbeste.com>
0b025f17 - 2022-07-31 - PRATYAY MUSTAFI <pratyaymustafi@gmail.com>
2022-08-05 07:18:24 +00:00
dependabot[bot] 27cdbeb9c4
Bump pycodestyle from 2.8.0 to 2.9.1
Bumps [pycodestyle](https://github.com/PyCQA/pycodestyle) from 2.8.0 to 2.9.1.
- [Release notes](https://github.com/PyCQA/pycodestyle/releases)
- [Changelog](https://github.com/PyCQA/pycodestyle/blob/main/CHANGES.txt)
- [Commits](https://github.com/PyCQA/pycodestyle/compare/2.8.0...2.9.1)

---
updated-dependencies:
- dependency-name: pycodestyle
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-05 07:06:16 +00:00
dependabot[bot] c6b5bc5c69
Bump setproctitle from 1.3.0 to 1.3.1
Bumps [setproctitle](https://github.com/dvarrazzo/py-setproctitle) from 1.3.0 to 1.3.1.
- [Release notes](https://github.com/dvarrazzo/py-setproctitle/releases)
- [Changelog](https://github.com/dvarrazzo/py-setproctitle/blob/master/HISTORY.rst)
- [Commits](https://github.com/dvarrazzo/py-setproctitle/compare/version-1.3.0...version-1.3.1)

---
updated-dependencies:
- dependency-name: setproctitle
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-05 07:06:12 +00:00
Markus Heiser 225f8caaf9
Merge pull request #1608 from allendema/brave-time_range
[enh] time range support for brave
2022-08-04 06:48:50 +02:00
Allen 58369e41d6 [wip] brave time range support 2022-08-04 04:47:25 +02:00
Markus Heiser 7c9c112484
Merge pull request #1560 from return42/http-accept-language
[mod] add 'Accept-Language' HTTP header to online processores
2022-08-01 17:59:49 +02:00
Markus Heiser 8df1f0c47e [mod] add 'Accept-Language' HTTP header to online processores
Most engines that support languages (and regions) use the Accept-Language from
the WEB browser to build a response that fits to the language (and region).

- add new engine option: send_accept_language_header

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 17:01:59 +02:00
Markus Heiser a2badb4fe4 [doc] add description of method EngineProcessor.get_params()
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-08-01 16:42:33 +02:00
Markus Heiser 1fbb514a4e
Merge pull request #1595 from tiekoetter/fix-uwsgi-buffer-size
[fix] uWSGI: increase buffer-size
2022-07-31 13:13:54 +02:00
Markus Heiser 48968bf46a [doc] list of changes that affect the infrastructure
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-07-31 13:08:06 +02:00
Léon Tiekötter 2b94fef7ec [fix] uWSGI: increase buffer-size
Increase max size of a request, by default it is 4k [1].  4096 as buffer-size is
too small and will result in the preference urls not working.

[1] https://uwsgi-docs.readthedocs.io/en/latest/Options.html#buffer-size
2022-07-31 12:40:06 +02:00