The error only occurs in the CI action, which is why we need to commit the coded
debug. As soon as the bug is identified and fixed, this commit can be reverted
/ the ``set -x`` can be removed from the code.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since #5280 has been merged, msgspec, like yaml, is a fixed part of the SearXNG
*settings framework* and therefore, like yaml, must be installed in the virtual
environment before installing SearXNG (``searx``).
The actual reason is that in SearXNG we store settings in the configuration that
are required for the installation of the ``searx`` package. This means that
these settings (from settings.yml) are read in during the installation, and all the
necessary tools for this (pyyaml, msgspec, setuptools, etc.) must be installed
beforehand (chicken or the egg dilemma).
Related:
- https://github.com/searxng/searxng/pull/5280
- https://github.com/searxng/searxng/pull/5254
Closes: https://github.com/searxng/searxng/issues/5343
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Mise en place config [1] does no longer support ``.tool-versions``
compatibility syntax, migrate to TOML ``mise.toml``.
In ``utils/lib_sxng_node.sh`` the node version was not updated, update to
24.3.0 (compare ``mise.toml``).
[1] https://mise.jdx.dev/configuration.html
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Docker buildx outputs the following error:
variable expansion is not supported for --from, define a new stage with FROM
using ARG from global scope as a workaround.
Also force BuildKit extension to be installed, legacy build is no longer
supported.
Closes https://github.com/searxng/searxng/issues/5219
This commit replaces `pip` in container builds with `uv` pip compat
with a 1:1 parity. The only thing that changes is the installation speed of the
wheels, which seems to be considerably faster, although I haven't been able to
properly quantify this yet.
uv also gives us more tools to manage the cache. We can revert the prior cache
changes in `container.yml` as we won't have duplicated wheels anymore.
pyrightconfig.json :
for the paths searx, searxng_extra and tests, individual rules were
defined (for example, in test fewer / different rules are needed than in the
searx package
searx/engines/__builtins__.pyi :
The builtin types that are added to the global namespace of a module by the
intended monkey patching of the engine modules / replaces the previous
filtering of the stdout using grep.
test.pyright_modified (utils/lib_sxng_test.sh) :
static type check of local modified files not yet commited
make test :
prerequisite 'test.pyright' has been replaced by 'test.pyright_modified'
searx/engines/__init__.py, searx/enginelib/__init__.py :
First, minimal typifications that were considered necessary.
Set minor versioning for most of the packages that iterate fast or we know won't
cause problems, and fixed versioning for the rest.
Packages going into bundles should be placed in "dependencies".
The inspection of prod bundles is necessary. Although it does not make a lot of
sense right now, it will be useful in later PR and will give us a reference to
start with.
TypeScript is a superset of JavaScript, converting the entire theme to
TypeScript allows us to receive much more feedback on possible issues made in
package updates or our own typos, furthermore, it allows to transpile properly
to lower specs. This PR couldn't be done in smaller commits, a lot of work
needed to make everything *work properly*:
- A browser baseline has been set that requires minimum **Chromium 93, Firefox
92 and Safari 15** (proper visuals/operation on older browser versions is not
guaranteed)
- LightningCSS now handles minification and prefix creation for CSS.
- All hardcoded polyfills and support for previous browser baseline versions
have been removed.
- Convert codebase to TypeScript.
- Convert IIFE to ESM, handling globals with IIFE is cumbersome, ESM is the
standard for virtually any use of JS nowadays.
- Vite now builds the theme without the need for `vite-plugin-static-copy`.
- `searxng.ready` now accepts an array of conditions for the callback to be
executed.
- Replace `leaflet` with `ol` as there were some issues with proper Vite
bundling.
- Merged `head` with `main` script, as head was too small now.
- Add `assertElement` to properly check the existence of critical DOM elements.
- `searxng.on` renamed to `searxng.listen` with some handling improvements.
The HTTP X-Forwarded-Proto (XFP) request header is a *de-facto* standard header
for identifying the protocol (HTTP or HTTPS) that a client used to connect to a
proxy or load balancer.[1]
The ``X-Scheme`` header was added 10 years ago, why ``X-Scheme`` was used back
then and not ``X-Forwarded-Proto``, nobody knows today / possibly because
``X-Forwarded-Proto`` wasn't a *de-facto* standard back then.
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-Proto
[2] https://github.com/searx/searx/commit/6ef7c3276
Building the container currently does not work properly.
When rebuilding several times with `make container`, `version_frozen.py`
is recreated, which wouldn't be an issue if the file’s timestamp was constant.
Now, when creating `version_frozen.py`, it will have the same timestamp as the
commit when it was created. (`version_frozen.py` is moved to a dedicated layer).
Reusing "builder" cache when building "dist" could be slow
(CD reports 2 seconds, but locally I've seen it take up to 10 seconds),
so the Dockerfile is now split and we save a couple steps
by importing the "builder" image directly.
The last changes made it possible to remove the layer cache in "builder",
since the overhead is now greater than building the layers from scratch.
Until now, all "dist" layers were squashed into a single layer,
which in most cases is a good idea
(except for storage/delivery pricing/overhead), but in our case,
since we manage the entire pipeline, we can ignore this
and share layers between builds.
This means (for example) that if we change files unrelated to the container
in several consecutive commits (documentation changes), we don't have to push
the entire image to registry, but only the different layers
(`version_frozen.py` in this example).
The same applies when pulling, as only the layers that have changed
compared to the local layers will be downloaded (that's the theory,
we'll see if this works as expected or if we need to tweak something else).
With this change, the "latest" tag will be visually higher (on registry tag list). Right now, it appears under the "DOCKER_TAG" manifest tag, which can be confusing.
Pyright has been installed twice so far, once via `package.json` and once in the
test script via `npx --no-install`. Separating the type checks in the CI and on
the developer desktop is also not necessary.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Implement rules and functions to format shell scripts:
$ make format.shell
or alternatively to format all source code:
$ make format
The formatter `shfmt` reads the rules from the editorconfig[1]
If any EditorConfig files are found, they will be used to apply formatting
options. If any parser or printer flags are given to the tool, no
EditorConfig files will be used.
[1] https://github.com/patrickvane/shfmt?tab=readme-ov-file#description
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
While looking at ways to better handle static files, I saw a package that replaces Flask `static_folder` functionality. Not only it's considerably faster, but already includes the capability to serve sidecars without having to intercept. This also replaces the uWSGI folder mapping functionality.
Closes https://github.com/searxng/searxng/issues/4977
This is one of various PR to refactor the simple theme internally.
Replace eslint tool with Biome. I have been using this for quite some time, and
it will help us to have more consistent and valid code without extending on
other third party plugins.
Removes unused dependencies.
This is a poorly designed instruction, which is hardcoded and cannot be easily modified or maintained on a rolling release sw like ours. This *should* be set in the SearXNG Docker Compose template, not in the image itself.
The OCI format is now used since we no longer have the HEALTHCHECK on the Dockerfile.
Closes https://github.com/searxng/searxng/issues/4906
Closes https://github.com/searxng/searxng/issues/4722
I'm not too pleased to reverse this, but issues like https://github.com/searxng/searxng/issues/4792 have not been foreseen, and we can't just turn away. It has become apparent over the last weeks that there are still quite a few people with an incompatible CPU or having SearXNG on some random VM provider who can't (or won't) modify the configuration of their machines to expose the features needed for x86_64v2 march.
As I don't want to trash the work with apko and base images, I thought about trying building Alpine again now that we have all the container related workflow refactored.
There will still be the discussion of whether to use musl and its drawbacks, but right now I don't know any other alternatives.
The nice part of this is that both Dockerfiles (mainline and legacy) can now be unified under the same umbrella again.
Closes https://github.com/searxng/searxng/issues/4792
Closes https://github.com/searxng/searxng/issues/4753
That entrypoint is prone to screw things up, especially with permission handling. The new script handles initialization better and fixes some issues like delayed settings update via ENVs and timestamp overwriting, also adjusts what should be copied into the container.
Related https://github.com/searxng/searxng/pull/4721#issuecomment-2850272129
Currently, we have 1100~ cache images uploaded to GHCR that weigh more than 300 MB each (most of them are layers from the second phase of the Dockerfile that were uploaded by mistake, read below). To avoid problems, I have set up a new job in a new workflow to be run weekly purging all images older than 1 week, but leaving always the 100 most recent ones.
Only the builder images should be uploaded to cache, the actual behaviour not only slows down the time for building the container, but also wastes lots of space by saving large and useless layers to GHCR that will never be used again.
container.yml will run after integration.yml COMPLETES successfully and in master branch.
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows.
* Podman is now supported to build the container images (Docker also received a refactor, merging both build and buildx)
* Container images are being built by Buildah instead of Docker BuildKit.
* Container images are tested before release.
* Splitting "modern" (amd64 & arm64) and "legacy" (armv7) arches on different Dockerfiles allowing future optimizations.
The big queries for initializing and updating the currencies take longer than
the default of the wikidata engine, which is only 3sec.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch implements the template generation of the template:
searx/templates/simple/icons.html
by the way the icon set (the icon names) has been normalized:
film-outline --> film
magnet-outline --> magnet
..
warning --> alert
Some missing (categorie) icons had been added.
Some of the ionicons are not suitable for a dark theme, we fixed the svg
manually in src/svg/ionicons:
- https://github.com/searxng/searxng/pull/4284#issuecomment-2680550342
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Vite is a build tool, it consists of two major parts [1]:
- A dev server that provides rich feature enhancements over native ES modules,
for example extremely fast Hot Module Replacement (HMR).
- A build command that bundles your code with Rollup, pre-configured to output
highly optimized static assets for production.
[1] https://vite.dev/guide/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
