Favicons (source)

Implementations for providing the favicons in SearXNG

searx.favicons.favicon_proxy()

REST API of SearXNG’s favicon proxy service

/favicon_proxy?authority=<...>&h=<...>

authority:

Domain name RFC 3986 / see favicon_url

h:

HMAC RFC 2104, build up from the server.secret_key setting.

searx.favicons.favicon_url(authority: str) → str

Function to generate the image URL used for favicons in SearXNG’s result lists. The authority argument (aka netloc / RFC 3986) is usually a (sub-) domain name. This function is used in the HTML (jinja) templates.

<div class="favicon">
   <img src="{{ favicon_url(result.parsed_url.netloc) }}">
</div>

The returned URL is a route to favicon_proxy REST API.

If the favicon is already in the cache, the returned URL is a data URL (something like data:image/png;base64,...). By generating a data url from the cache.FaviconCache, additional HTTP roundtripps via the favicon_proxy are saved. However, it must also be borne in mind that data urls are not cached in the client (web browser).

Favicons Config

class searx.favicons.config.FaviconConfig(cfg_schema: int, cache: ~searx.favicons.cache.FaviconCacheConfig = <class 'searx.favicons.cache.FaviconCacheConfig'>, proxy: ~searx.favicons.proxy.FaviconProxyConfig = <class 'searx.favicons.cache.FaviconCacheConfig'>)

The class aggregates configurations of the favicon tools

classmethod from_toml_file(cfg_file: Path, use_cache: bool) → FaviconConfig

Create a config object from a TOML file, the use_cache argument specifies whether a cache should be used.

cache: FaviconCacheConfig

Setup of the cache.FaviconCacheConfig.

cfg_schema: int

Config’s schema version. The specification of the version of the schema is mandatory, currently only version CONFIG_SCHEMA is supported. By specifying a version, it is possible to ensure downward compatibility in the event of future changes to the configuration schema

proxy: FaviconProxyConfig

Setup of the proxy.FaviconProxyConfig.

searx.favicons.config.CONFIG_SCHEMA: int = 1

Version of the configuration schema.

searx.favicons.config.TOML_CACHE_CFG: dict[str, FaviconConfig] = {}

Cache config objects by TOML’s filename.

Favicons Proxy

Implementations for a favicon proxy

class searx.favicons.proxy.FaviconProxyConfig(max_age: int = 604800, secret_key: str = '', resolver_timeout: int = 3.0, resolver_map: dict[str, str] = <factory>, favicon_path: str = '/home/runner/work/searxng/searxng/searx/static/themes/{theme}/img/empty_favicon.svg', favicon_mime_type: str = 'image/svg+xml')

Configuration of the favicon proxy.

favicon(**replacements)

Returns pathname and mimetype of the default favicon.

favicon_data_url(**replacements)

Returns data image URL of the default favicon.

get_resolver(name: str) → Callable | None

Returns the callable object (function) of the resolver with the name. If no resolver is registered for the name, None is returned.

max_age: int

HTTP header Cache-Control max-age

resolver_map: dict[str, str]

The resolver_map is a key / value dictionary where the key is the name of the resolver and the value is the fully qualifying name (fqn) of resolver’s function (the callable). The resolvers from the python module searx.favicons.resolver are available by default.

resolver_timeout: int

Timeout which the resolvers should not exceed, is usually passed to the outgoing request of the resolver. By default, the value from outgoing.request_timeout setting is used.

secret_key: str

By default, the value from server.secret_key setting is used.

searx.favicons.proxy.favicon_proxy()

REST API of SearXNG’s favicon proxy service

/favicon_proxy?authority=<...>&h=<...>

authority:

Domain name RFC 3986 / see favicon_url

h:

HMAC RFC 2104, build up from the server.secret_key setting.

searx.favicons.proxy.favicon_url(authority: str) → str

Function to generate the image URL used for favicons in SearXNG’s result lists. The authority argument (aka netloc / RFC 3986) is usually a (sub-) domain name. This function is used in the HTML (jinja) templates.

<div class="favicon">
   <img src="{{ favicon_url(result.parsed_url.netloc) }}">
</div>

The returned URL is a route to favicon_proxy REST API.

If the favicon is already in the cache, the returned URL is a data URL (something like data:image/png;base64,...). By generating a data url from the cache.FaviconCache, additional HTTP roundtripps via the favicon_proxy are saved. However, it must also be borne in mind that data urls are not cached in the client (web browser).

searx.favicons.proxy.search_favicon(resolver: str, authority: str) → tuple[None | bytes, None | str]

Sends the request to the favicon resolver and returns a tuple for the favicon. The tuple consists of (data, mime), if the resolver has not determined a favicon, both values are None.

data:

Binary data of the favicon.

mime:

Mime type of the favicon.

Favicons Resolver

Implementations of the favicon resolvers that are available in the favicon proxy by default. A resolver is a function that obtains the favicon from an external source. The resolver function receives two arguments (domain, timeout) and returns a tuple (data, mime).

searx.favicons.resolvers.allesedv(domain: str, timeout: int) → tuple[None | bytes, None | str]

Favicon Resolver from allesedv.com / https://favicon.allesedv.com/

searx.favicons.resolvers.duckduckgo(domain: str, timeout: int) → tuple[None | bytes, None | str]

Favicon Resolver from duckduckgo.com / https://blog.jim-nielsen.com/2021/displaying-favicons-for-any-domain/

searx.favicons.resolvers.google(domain: str, timeout: int) → tuple[None | bytes, None | str]

Favicon Resolver from google.com

searx.favicons.resolvers.yandex(domain: str, timeout: int) → tuple[None | bytes, None | str]

Favicon Resolver from yandex.com

Favicons Cache

Implementations for caching favicons.

FaviconCacheConfig:

Configuration of the favicon cache

FaviconCache:

Abstract base class for the implementation of a favicon cache.

FaviconCacheSQLite:

Favicon cache that manages the favicon BLOBs in a SQLite DB.

FaviconCacheNull:

Fallback solution if the configured cache cannot be used for system reasons.

---

class searx.favicons.cache.FaviconCache(cfg: FaviconCacheConfig)

Abstract base class for the implementation of a favicon cache.

abstract maintenance(force=False)

Performs maintenance on the cache

abstract set(resolver: str, authority: str, mime: str | None, data: bytes | None) → bool

Set data and mime-type in the cache. If data is None, the FALLBACK_ICON is registered. in the cache.

abstract state() → FaviconCacheStats

Returns a FaviconCacheStats (key/values) with information on the state of the cache.

class searx.favicons.cache.FaviconCacheConfig(db_type: Literal['sqlite', 'mem'] = 'sqlite', db_url: Path = PosixPath('/tmp/faviconcache.db'), HOLD_TIME: int = 2592000, LIMIT_TOTAL_BYTES: int = 52428800, BLOB_MAX_BYTES: int = 20480, MAINTENANCE_PERIOD: int = 3600, MAINTENANCE_MODE: Literal['auto', 'off'] = 'auto')

Configuration of the favicon cache.

BLOB_MAX_BYTES: int

The maximum BLOB size in bytes that a favicon may have so that it can be saved in the cache. If the favicon is larger, it is not saved in the cache and must be requested by the client via the proxy.

HOLD_TIME: int

Hold time (default in sec.), after which a BLOB is removed from the cache.

LIMIT_TOTAL_BYTES: int

Maximum of bytes (default) stored in the cache of all blobs. Note: The limit is only reached at each maintenance interval after which the oldest BLOBs are deleted; the limit is exceeded during the maintenance period. If the maintenance period is too long or maintenance is switched off completely, the cache grows uncontrollably.

MAINTENANCE_MODE: Literal['auto', 'off']

Type of maintenance mode

auto:

Maintenance is carried out automatically as part of the maintenance intervals (MAINTENANCE_PERIOD); no external process is required.

off:

Maintenance is switched off and must be carried out by an external process if required.

MAINTENANCE_PERIOD: int

Maintenance period in seconds / when MAINTENANCE_MODE is set to auto.

db_type: Literal['sqlite', 'mem']

Type of the database:

sqlite:

cache.FaviconCacheSQLite

mem:

cache.FaviconCacheMEM (not recommended)

db_url: Path

URL of the SQLite DB, the path to the database file.

class searx.favicons.cache.FaviconCacheMEM(cfg)

Favicon cache in process’ memory. Its just a POC that stores the favicons in the memory of the process.

Attention

Don’t use it in production, it will blow up your memory!!

maintenance(force=False)

Performs maintenance on the cache

set(resolver: str, authority: str, mime: str | None, data: bytes | None) → bool

Set data and mime-type in the cache. If data is None, the FALLBACK_ICON is registered. in the cache.

state()

Returns a FaviconCacheStats (key/values) with information on the state of the cache.

class searx.favicons.cache.FaviconCacheNull(cfg: FaviconCacheConfig)

A dummy favicon cache that caches nothing / a fallback solution. The NullCache is used when more efficient caches such as the FaviconCacheSQLite cannot be used because, for example, the SQLite library is only available in an old version and does not meet the requirements.

maintenance(force=False)

Performs maintenance on the cache

set(resolver: str, authority: str, mime: str | None, data: bytes | None) → bool

Set data and mime-type in the cache. If data is None, the FALLBACK_ICON is registered. in the cache.

state()

Returns a FaviconCacheStats (key/values) with information on the state of the cache.

class searx.favicons.cache.FaviconCacheSQLite(cfg: FaviconCacheConfig)

Favicon cache that manages the favicon BLOBs in a SQLite DB. The DB model in the SQLite DB is implemented using the abstract class sqlitedb.SQLiteAppl.

The following configurations are required / supported:

• FaviconCacheConfig.db_url

• FaviconCacheConfig.HOLD_TIME

• FaviconCacheConfig.LIMIT_TOTAL_BYTES

• FaviconCacheConfig.BLOB_MAX_BYTES

• MAINTENANCE_PERIOD

• MAINTENANCE_MODE

maintenance(force=False)

Performs maintenance on the cache

set(resolver: str, authority: str, mime: str | None, data: bytes | None) → bool

Set data and mime-type in the cache. If data is None, the FALLBACK_ICON is registered. in the cache.

state() → FaviconCacheStats

Returns a FaviconCacheStats (key/values) with information on the state of the cache.

DB_SCHEMA: int = 1

As soon as changes are made to the DB schema, the version number must be increased. Changes to the version number require the DB to be recreated (or migrated / if an migration path exists and is implemented).

DDL_BLOBS = 'CREATE TABLE IF NOT EXISTS blobs (\n  sha256     TEXT,\n  bytes_c    INTEGER,\n  mime       TEXT NOT NULL,\n  data       BLOB NOT NULL,\n  PRIMARY KEY (sha256))'

Table to store BLOB objects by their sha256 hash values.

DDL_BLOB_MAP = "CREATE TABLE IF NOT EXISTS blob_map (\n    m_time     INTEGER DEFAULT (strftime('%s', 'now')),  -- last modified (unix epoch) time in sec.\n    sha256     TEXT,\n    resolver   TEXT,\n    authority  TEXT,\n    PRIMARY KEY (resolver, authority))"

Table to map from (resolver, authority) to sha256 hash values.

SQL_DROP_LEFTOVER_BLOBS = 'DELETE FROM blobs WHERE sha256 IN ( SELECT b.sha256   FROM blobs b   LEFT JOIN blob_map bm     ON b.sha256 = bm.sha256  WHERE bm.sha256 IS NULL)'

Delete blobs.sha256 (BLOBs) no longer in blob_map.sha256.

property next_maintenance_time: int

Returns (unix epoch) time of the next maintenance.

class searx.favicons.cache.FaviconCacheStats(favicons: int | None = None, bytes: int | None = None, domains: int | None = None, resolvers: int | None = None)

Dataclass wich provides information on the status of the cache.

searx.favicons.cache.init(cfg: FaviconCacheConfig)

Initialization of a global CACHE

searx.favicons.cache.maintenance(force: bool = True, debug: bool = False)

perform maintenance of the cache

searx.favicons.cache.state()

show state of the cache