91–99 of 99 posts
Pretty urls are the most unnecessary thing that was invented, they don't provide anything that non pretty urls can't provide and millions of parsers have to process them on every request. What a waste.
What is a "pretty url"?
Okay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.
This system seems pretty weird to me. I was wondering, can that clash with a "normal" domain registered as "xn--....."? Apparently there is another specific rule in RFC 5891 saying "The Unicode string MUST NOT contain "--" (two consecutive hyphens) in the third and fourth character positions" [0] Also, if I was forced to represent Unicode as ASCII, punycode encoding is not the obvious one - it's pretty confusing. But…
Actually, I wonder what happens if you take a "normal" (i.e. non-IDN, ascii-only) domain and encode it as Punycode. Should the encoded and non-encoded domains be considered identical or separate? (for purposes of DNS resolutions, origin separation, etc)
Identical would be more intuitive and would match the behavior of domain names with non-ascii characters - on the other hand, this would require reworking of ALL non-punycode-aware DNS software, which I'm doubtful is possible.
So this seems like a tricky thing to get right.
No mention of prior art? The Hyperlink library has stated correctness as its goal for a long time: https://pypi.org/project/hyperlink Of course there is always room for new projects, but it still feels weird to act as if this is the first time anybody has ever tried this. It seems like a lot of people are under this same mistaken impression, at least according to the sample of HN users who commented in this thread.
Hyperlink (which I didn't know existed, by the way) is not a parser for the WHATWG spec, it's for RFC3986. You seem to be getting things confused.
Earlier quoted context omitted.
This system seems pretty weird to me. I was wondering, can that clash with a "normal" domain registered as "xn--....."? Apparently there is another specific rule in RFC 5891 saying "The Unicode string MUST NOT contain "--" (two consecutive hyphens) in the third and fourth character positions" [0] Also, if I was forced to represent Unicode as ASCII, punycode encoding is not the obvious one - it's pretty confusing. But…
IDNs and Punycode were basically bolted-on extensions to DNS that were added after DNS was already widely deployed. Because there was no "proper" extension mechanism available, it was a design requirement that they can be implemented "on top" of the standard DNS without having to change any of the underlying components. So I think most of the DNS infrastructure can be (and is) still completely unaware that IDNs and P…
>>> "foo".encode("idna")
b'foo'
>>> "fooé".encode("idna")
b'xn--foo-dma'
There's also the "punycode" encoding available, but that does something subtly different that's not quite how domains get encoded:
>>> "foo".encode("punycode")
b'foo-'
>>> "fooé".encode("punycode")
b'foo-dma'Honestly to hell with the WHATWG's weird pseudo-standard. Backslashes? Five forward slashes? No one is sending those URLs around, and if they are, they should fix it. The only thing that needs to deal with that kind of malformed URL is the browser's address bar. If browsers want to do fixups on broken URLs at that point, they should feel free to, but it shouldn't pollute an entire spec. (It's not even a real standard…
It's always been extremely funny to me how arguments like this and from the curl author go. "Yes, I had to change curl away from strictly accepting two slashes, for web compatibility. But `while(slash) { advance_parser() }`? That's completely unreasonable! `while(slash && ++i As for your claim about living standards, I'd encourage you to read https://whatwg.org/faq#living-standard
They seem to be making reference to things like the RFC system, but those get updated too.
Boom, you have now parsed a URL.
Wanna parse an http query params into host,port, resource? Speak appropriately, ask how to parse an http request, wanna parse a resource? Get into those semantics, be precise
Earlier quoted context omitted.
IDNs and Punycode were basically bolted-on extensions to DNS that were added after DNS was already widely deployed. Because there was no "proper" extension mechanism available, it was a design requirement that they can be implemented "on top" of the standard DNS without having to change any of the underlying components. So I think most of the DNS infrastructure can be (and is) still completely unaware that IDNs and P…
Python has idna-encoding built in these days, so I figured I'd do a quick check to see what happens: >>> "foo".encode("idna") b'foo' >>> "fooé".encode("idna") b'xn--foo-dma' So indeed a punycode'd ascii domain would remain unchanges by the looks of it. There's also the "punycode" encoding available, but that does something subtly different that's not quite how domains get encoded: >>> "foo".encode("punycode") b'foo-'…
https://docs.python.org/3.12/library/codecs.html#module-enco...
The recommend the 3rd party 'idna' module for this:
https://pypi.org/project/idna/
IDNA 2003 is a particular annoyance of mine: The IDNA 2003 algorithm didn't encode the german 'ß' character, or rather 'wrongly', through overeager use of Unicode normalisation in the nameprep part. Then the browser makers for a long time stood still and didn't upgrade to IDNA 2008, which fixed that bug among other things. The WhatWG in its self-appointed role as stenograph of the browser cartel didn't change its weird URL spec. But that seems to have changed in recent years. Of course the original sin of IDNA was making it client-side. :/