- fix the issue of fetching more the 7000 *languages*
- improve the request function and filter by language & country
- implement time_range_support & safesearch
- add more fields to the response from dailymotion (allow_embed, length)
- better clean up of HTML tags in the 'content' field.
This is more or less a complete rework based on the '/videos' API from [1].
This patch cleans up the language list in SearXNG that has been polluted by the
ISO-639-3 2 and 3 letter codes from dailymotion languages which have never been
used.
[1] https://developers.dailymotion.com/tools/
Closes: https://github.com/searxng/searxng/issues/1065
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch implements a bolierplate to share content from info-pages of the
SearXNG instance (URL /info) with the project documentation (path /docs/user).
The info pages are using Markdown (CommonMark), to include them in the project
documentation (reST) the myst-parser [1] is used in the Sphinx-doc build chain.
If base_url is known (defined in settings.yml) links to the instance are also
inserted into the project documentation::
searxng_extra/docs_prebuild
[1] https://www.sphinx-doc.org/en/master/usage/markdown.html
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* drop image_layout.js from simple theme
* move image_layout.js to oscar theme and delete common js dir (since its empty now)
* align top position of image detail modal with bottom position of search header
* use flexbox to display images; row height can be set via @results-image-row-height in defenitions.less
* display span title underneath each image with a max width of 12rem
* increase margin and padding around image article on desktop and tablet
* make article height smaller on phone layout (height of 6rem) to display more content on current view
* remove content from result, if the title and content matches
* use a group that cotains the flex image article, if images are mixed with other categories
* fix pylint issues in webapp.py
* use the default.html result template in unit tests (thanks @return42)
Now that about.html extends page_with_header.html
it already has a link to the start page and removing
the link makes it easier to extract the page title
from the Markdown for the following commit.
Bangs with a `*` suffix (e.g. `!!d*`) overwrite Bangs with the same
prefix (e.g. `!!d`) [1]. This can be avoid when a non printable character is
used to tag a LEAF_KEY.
[1] https://github.com/searxng/searxng/pull/740#issuecomment-1010411888
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The previous implementation used two hash sets and a list.
... that's not necessary ... a single hash map suffices.
And it's also less error prone ... because the previous data structure
allowed a setting to be enabled and disabled at the same time.
The ? search operator has been broken for some time and
currently only raises the question why it's still there.
## Context ##
The query "Paris !images" searches for "Paris" in the "images" category.
Once upon a time Searx supported "Paris ?images" to search for "Paris"
in the currently enabled categories and the "images" category.
The feature makes sense ... the ? syntax does not.
We will hopefully introduce a +!images syntax in the future.
Fixes#702.
The tab icon names are currently hard coded in the templates.
This commit lets us introduce an icon property in the future, e.g:
categories_as_tabs:
general:
icon: search-outline
Previously all categories were displayed as search engine tabs.
This commit changes that so that only the categories listed under
categories_as_tabs in settings.yml are displayed.
This lets us introduce more categories without cluttering up the UI.
Categories not displayed as tabs can still be searched with !bangs.
Fix pylint issues from commit (3d96a983)
[format.python] initial formatting of the python code
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Disable the python code formatting from python-black, where the readability of
code suffers by formatting.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
If there is no write access, there is no need for global. Remove global
statement if there is no assignment.
global-variable-not-assigned:
Using global for names but no assignment is done Used when a variable is
defined through the "global" statement but no assignment to this variable is
done.
In Pylint 2.11 the global-variable-not-assigned checker now catches global
variables that are never reassigned in a local scope and catches (reassigned)
functions [1][2]
[1] https://pylint.pycqa.org/en/latest/whatsnew/2.11.html
[2] https://github.com/PyCQA/pylint/issues/1375
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In unit tests settings from
searx/settings.yml
and the user settings from:
unit/settings/test_settings.yml
are used. In the latter, settings can be activated that are needed in the unit
test but should not activated by default in production.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add a new environment variable SEARX_DISABLE_ETC_SETTINGS
to disable loading of /etc/searx/settings.yml
unit tests:
* set SEARX_DISABLE_ETC_SETTINGS to 1
* remove SEARX_SETTINGS_PATH if it exists
tests/units/network/test_network.py requires a call to searx.network.network.initialize
Depending of the test order execution, this function was sometimes call in another test,
sometimes not.
This commit ensure there is a call to initialize()
it prepares the new architecture change,
everything about multithreading in moved in the searx.search.* packages
previously the call to the "init" function of the engines was done in searx.engines:
* the network was not set (request not sent using the defined proxy)
* it requires to monkey patch the code to avoid HTTP requests during the tests
settings.yml:
* outgoing.networks:
* can contains network definition
* propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections,
keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries
* retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time)
* local_addresses can be "192.168.0.1/24" (it supports IPv6)
* support_ipv4 & support_ipv6: both True by default
see https://github.com/searx/searx/pull/1034
* each engine can define a "network" section:
* either a full network description
* either reference an existing network
* all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
Some of our interface locales include uppercase country codes,
which are separated by `_` instead of the more common `-`.
Also, a browser's `Accept-Language` header could be in lowercase.
This commit attempts to normalize those cases so a browser's
language+country codes can better match with our locales.
This solution assumes that our UI locales have nothing more than
language and optionally country. If we ever add a script specific
locale like `zh-Hant-TW` this would have to change to accomodate
that, but the idea would be pretty much the same as this fix.
from_bang is True when the user query contains a bang.
In this case the category is also set to 'none'.
from_bang only usage was in searx.webadapter.parse_specific :
if from_bang is True, then the EngineRef category is ignored and force to 'none'.
This commit also removes the searx.webadapter.parse_sepecific function.
see searx.search.processors.abstract.EngineProcessor
First the method searx call the get_params method.
If the return value is not None, then the searx call the method search.
If test_engines_init.py runs before test_standalone_searx.py, the engine list is not empty.
It makes test_get_search_query flaky.
This commit initializes the engline list in test_standalone_searx.py
use
from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url # NOQA
so it is possible to easily remove all unused import using autoflake:
autoflake --in-place --recursive --remove-all-unused-imports searx tests
* URL / : the index page displayed the selected or the default category.
* URL / : when the q parameter is set using the URL, the redirect includes the URL query.
* URL /search : an empty query doesn't raise an exception.
This makes it easier to separately handle search and index requests
from a web server or from a reverse proxy.
If a request to index contains a query, a permanent redirect HTTP response
is returned. This should give some level of backwards compatibility
for users that have set a searx instance in their browser's search bar.
Xpath engine and results template changed to account for the fact that
archive.org doesn't cache .onions, though some onion engines migth have
their own cache.
Disabled by default. Can be enabled by setting the SOCKS proxies to
wherever Tor is listening and setting using_tor_proxy as True.
Requires Tor and updating packages.
To avoid manually adding the timeout on each engine, you can set
extra_proxy_timeout to account for Tor's (or whatever proxy used) extra
time.
Markus Heiser
Alexandre Flament
mrpaulblack
Martin Fischer
Igor Rzegocki
Marc Abonce Seguin
Adam Tauber
Daniel Hones
rachmadaniHaryono
rachmadani haryono
a01200356