This patch adds GitHub Code Search [1] engine to allow querying the codebases.
Template code.html is changed to allow passthrough of strip and highlighting
options.
Engine Searchcode is adjusted to pass filename and not rely on hardcoded
extensions.
GitHub search code API does not return the exact code line indices, this
implementation assigns the code arbitrary numbers starting from 1
(effectively relabeling the code).
The API allows for unauth calls, and the default engine settings default to
that, although the calls are heavily rate limited.
The 'text' lexer is the default pygments lexer when parsing fails.
[1] https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#search-code
Co-authored-by: Markus Heiser <markus.heiser@darmarIT.de>
Wordnik is now an answerer and not in the infobox anymore: it uses the
translations answerer, because it provides all the features needed. By default,
only its first results is shown
Additionally a new "define" category is added - I know, it's the same as the
"dictionaries" category, but I don't think we can alias categories. This allows
to search e.g. for `!define tree`, the idea is to allow easy searches for
definitions of words.
Related:
- https://github.com/searxng/searxng/issues/4111
What's changed?
- this PR adds Pixabay, a collection of royalty free images
- additionaly it seems to have some videos, so there's an engine for it too
Author Notes
- when using SearXNG's transport, all our requests will get blocked, probably due to fingerprinting
- we should find an alternative solution because this is just a hacky change to make things work for now, but idk how ...
With the new implementation for removing tracker arguments from URLs, we use the
tracker definitions of the ClearURLs project [1]. These definitions are
regularly updated and significantly improve data protection, which is why we
should now also enable this plugin by default.
[1] https://github.com/ClearURLs
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Tube Archivist [1] is a self-hosted project which archives youtube videos on
your own local server. This engine connects with Tube Archivist's search API to
allow searching from SearXNG into your own hosted videos.
[1] https://www.tubearchivist.com/
Signed-off-by: Robert M. Clabough <robert@claobugh.tech>
Co-authored-by: Bnyro <bnyro@tutanota.com>
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Icons category makes sense because it allows to quickly search for free SVG
icons to use for websites / other designs with a quick `!icons` query
Icons don't seem to fit into the normal images category that well because icons
are quite a special type of images
Re-writes the Mullvad Leta integration to work with the new breaking changes.
Mullvad Leta is a search engine proxy. Currently Leta only offers text search
results not image, news or any other types of search result. Leta acts as a
proxy to Google and Brave search results.
- Remove docstring comments regarding requiring the use of Mullvad VPN, which is
no longer a hard requirement.
- configured two engines: ``mullvadleta`` (uses google) and
``mullvadleta brave`` (uses brave)
- since leta may not provide up-to-date search results, both search engines are
disabled by default.
.. hint::
Leta caches each search for up to 30 days. For example, if you use search
terms like ``news``, contrary to your intention you'll get very old results!
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Signed-off-by: Grant Lanham <contact@grantlanham.com>
This patch brings two major changes:
- ``Result.filter_urls(..)`` to pass a filter function for URL fields
- The ``enabled_plugins:`` section in SearXNG's settings do no longer exists.
To understand plugin development compile documentation:
$ make docs.clean docs.live
and read http://0.0.0.0:8000/dev/plugins/development.html
There is no longer a distinction between built-in and external plugin, all
plugins are registered via the settings in the ``plugins:`` section.
In SearXNG, plugins can be registered via a fully qualified class name. A
configuration (`PluginCfg`) can be transferred to the plugin, e.g. to activate
it by default / *opt-in* or *opt-out* from user's point of view.
built-in plugins
================
The built-in plugins are all located in the namespace `searx.plugins`.
.. code:: yaml
plugins:
searx.plugins.calculator.SXNGPlugin:
active: true
searx.plugins.hash_plugin.SXNGPlugin:
active: true
searx.plugins.self_info.SXNGPlugin:
active: true
searx.plugins.tracker_url_remover.SXNGPlugin:
active: true
searx.plugins.unit_converter.SXNGPlugin:
active: true
searx.plugins.ahmia_filter.SXNGPlugin:
active: true
searx.plugins.hostnames.SXNGPlugin:
active: true
searx.plugins.oa_doi_rewrite.SXNGPlugin:
active: false
searx.plugins.tor_check.SXNGPlugin:
active: false
external plugins
================
SearXNG supports *external plugins* / there is no need to install one, SearXNG
runs out of the box.
- Only show green hosted results: https://github.com/return42/tgwf-searx-plugins/
To get a developer installation in a SearXNG developer environment:
.. code:: sh
$ git clone git@github.com:return42/tgwf-searx-plugins.git
$ ./manage pyenv.cmd python -m \
pip install -e tgwf-searx-plugins
To register the plugin in SearXNG add ``only_show_green_results.SXNGPlugin`` to
the ``plugins:``:
.. code:: yaml
plugins:
# ...
only_show_green_results.SXNGPlugin:
active: false
Result.filter_urls(..)
======================
The ``Result.filter_urls(..)`` can be used to filter and/or modify URL fields.
In the following example, the filter function ``my_url_filter``:
.. code:: python
def my_url_filter(result, field_name, url_src) -> bool | str:
if "google" in url_src:
return False # remove URL field from result
if "facebook" in url_src:
new_url = url_src.replace("facebook", "fb-dummy")
return new_url # return modified URL
return True # leave URL in field unchanged
is applied to all URL fields in the :py:obj:`Plugin.on_result` hook:
.. code:: python
class MyUrlFilter(Plugin):
...
def on_result(self, request, search, result) -> bool:
result.filter_urls(my_url_filter)
return True
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The engines now all use KeyValue results and return the results in a
EngineResults object.
The sqlite engine can return MainResult results in addition to KeyValue
results (based on engine's config in settings.yml),
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
