[fix] results: display internationalized domain names human-friendly

Some search engines encode domains that contain special characters
in the IDN format, i.e. by using Punycode (https://en.wikipedia.org/wiki/Punycode).

This means that domains formatted like that are not human readable
and thus very unintuive.

Also, as only some engines use Punycode, this often causes a result
to appear multiple times, once with a Punycode domain (e.g. xn--allestrungen-9ib.de),
and once with a normal domain (e.g. allestörungen.de).

Always formatting the domains nicely has the caveat that the official sites
are harder to distinguish from malicious clones that just swapped out some
characters by using Punycode, e.g. replaced `a` with `á`. I don't think
that will be much of an issue though - I don't think Punycode results
previously stopped users from clicking the link, and I also think that
most search engines filter out such bad results or don't even have them
indexed.
This commit is contained in:
Bnyro
2026-07-05 11:19:01 +02:00
parent 58e02a01ae
commit 9c49b7e0d7

View File

@@ -52,6 +52,12 @@ def _normalize_url_fields(result: "Result | LegacyResult"):
result.parsed_url = urllib.parse.urlparse(result.url)
if result.parsed_url:
# properly format special characters (e.g. "ä", "ö") in IDN domains
# e.g. "xn--strung-xxa.de" becomes "störung.de"
if result.parsed_url.netloc.startswith("xn--"):
netloc = result.parsed_url.netloc.encode().decode("idna")
result.parsed_url = result.parsed_url._replace(netloc=netloc)
result.parsed_url = result.parsed_url._replace(
# if the result has no scheme, use http as default
scheme=result.parsed_url.scheme or "http",