Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parsing URLs in Python

tkte.ch

Post reply on HN
Print view View original

51–60 of 99 posts

Re: Parsing URLs in Python

#51

Post by ekimekim »

VagabundoP wrote: post #30

Okay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.

To expand on the sibling comments: This encoding (called Punycode) works by combining the character to encode (é) and the position the character should be in (the 7th position out of a possible 7) into a single number. é is 233, there are 7 possible positions, and it is in position 6 (0-indexed) so that single number is 233 * 7 + 6 = 1637. This is then encoded via a fairly complex variable-length encoding scheme into the letters "fsa".

See https://en.wikipedia.org/wiki/Punycode#Encoding_the_non-ASCI...

Re: Parsing URLs in Python

#52

Post by kmike84 »

bormaj wrote: post #39

Why not just use `httpx`? If you're not bound to the stdlib, it's a great alternative to `requests` and url parse

The URL parsing in httpx is rfc3986, which is not the same as WHATWG URL living standard.

rfc3986 may reject URLs which browsers accept, or it can handle them in a different way. WHATWG URL living standard tries to put on paper the real browser behavior, so it's a much better standard if you need to parse URLs extracted from real-world web pages.

Re: Parsing URLs in Python

#55

Post by gjvc »

Solvency wrote: post #10
pyuser583 wrote: post #9

The Ada programming language is cursed with overlapping acronyms. GPS, ADA, SPARK, AWS. Seems it just got a little bit worse.

It is mindboggling to me how often developers create project names without even trying to search for precedent names in their own domain/industry . Calling this Ada is just ridiculous.

yes but they think it's punny/funny/clever

Re: Parsing URLs in Python

#56

Post by JulianWasTaken »

Nice.

I'll also throw in that I've recently wrote bindings to Mozilla's servo URL library.

Those live at https://github.com/crate-py/url

They're not complete yet (meaning only the parsing bits are exposed, not URL modification) but I too was frustrated with the state of URL parsing.

Re: Parsing URLs in Python

#57

Post by orf »

gjvc wrote: post #54
bormaj wrote: post #39

Why not just use `httpx`? If you're not bound to the stdlib, it's a great alternative to `requests` and url parse

httpx is great, and "needs to be in base"

No it doesn’t, absolutely not. It’s ironic that you say this after the post you’re commenting on spells out quite explicitly why things “in base” are hard to change and adapt.

Re: Parsing URLs in Python

#58

Post by phyzome »

Honestly to hell with the WHATWG's weird pseudo-standard. Backslashes? Five forward slashes? No one is sending those URLs around, and if they are, they should fix it. The only thing that needs to deal with that kind of malformed URL is the browser's address bar. If browsers want to do fixups on broken URLs at that point, they should feel free to, but it shouldn't pollute an entire spec.

(It's not even a real standard -- it's a "living standard", which means it just changes randomly and there's no way to actually say "yes, you're compliant with that".)

Re: Parsing URLs in Python

#59

Post by notso411 »

phyzome wrote: post #58

Honestly to hell with the WHATWG's weird pseudo-standard. Backslashes? Five forward slashes? No one is sending those URLs around, and if they are, they should fix it. The only thing that needs to deal with that kind of malformed URL is the browser's address bar. If browsers want to do fixups on broken URLs at that point, they should feel free to, but it shouldn't pollute an entire spec. (It's not even a real standard…

[dead]

Re: Parsing URLs in Python

#60

Post by andy99 »

js2 wrote: post #33
VagabundoP wrote: post #30

Okay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.

Because that's the Punycode representation: https://en.wikipedia.org/wiki/Punycode https://www.punycoder.com/

I wasn't aware of this, I'd seen those URLs before but only in the context of Chinese ones and thought it was Chinese-specific.

It's interesting because I just went down an apparent rabbit hole inplementing Byte-level encoding for using language models with unicode. There each byte in a unicode character is mapped to a printable character that goes up to 255 See https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c9...

And the actual list of characters:

https://github.com/rbitr/llm.f90/blob/dev/phi2/phi2/pretoken...

Post reply on HN