Earlier quoted context omitted.
urlib.parse is a pain. We really need something more like pathlib.Path.
There is https://github.com/gruns/furl
Yes! That’s the one I like!
Parsing URLs in Python
31–40 of 99 posts
Re: Parsing URLs in Python
#32Okay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.
It’s called an IDN. This is an encoding format called puny code that transforms international domains into ascii
Re: Parsing URLs in Python
#33Okay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.
Because that's the Punycode representation:
Re: Parsing URLs in Python
#34Re: Parsing URLs in Python
#35Hard to imagine the tradeoff of using a third party binary library developed this year vs just using urllib.parse being worth it. Is this solving a real problem?
urlib.parse is a pain. We really need something more like pathlib.Path.
You might be interested in https://github.com/fsspec/universal_pathlib
Re: Parsing URLs in Python
#36Earlier quoted context omitted.
Another post in this thread was downvoted and flagged (really?) for claiming that URL parsing isn't difficult. The linked article claims that "Parsing URLs correctly is surprisingly hard." As a software tester, I'm very willing to believe that, but I don't know that the article really made the case. I did find a paper describing some vulnerabilities in popular URL parsing libraries, including urllib and urllib3. Blog…
> If you remember the Log4j vulnerability from a couple of years ago, that was an URL parsing bug. I don't think that's a fair description of the issue. The log4j vulnerability was that it specifically added JNDI support ( https://issues.apache.org/jira/browse/LOG4J2-313 ) to property substitution ( https://logging.apache.org/log4j/2.x/manual/configuration.ht... ), which it would apply on logged messages. So it was a…
I didn't look into this in detail at the time, but the report's summary of CVE-2021-45046 is that the parser that validated an URL behaved differently than a separate parser used to fetch the URL, so an URL like
jndi:ldap://127.0.0.1#.evilhost.com:1389/a
Re: Parsing URLs in Python
#37Earlier quoted context omitted.
Not in the sense of differential vulnerabilities, since the standard library refuses to match any sort of modern standard. It's also 1. not a solo dev 2. Daniel Lemire 3. a serious engineering and research effort: https://arxiv.org/pdf/2311.10533.pdf
This is the commit history: https://github.com/TkTech/can_ada/commits/main/ I guess you are right that there are 2 commits from a different dev, so it is technically not a solo project. I still wouldn't ever use this in production code.
The can_ada repo threw me off, too. It looks super amateurish because of the lack of tests, fuzzers, etc.
But it appears that they've just exported the meat of the Ada project and left everything else upstream.
Re: Parsing URLs in Python
#38Earlier quoted context omitted.
Not in the sense of differential vulnerabilities, since the standard library refuses to match any sort of modern standard. It's also 1. not a solo dev 2. Daniel Lemire 3. a serious engineering and research effort: https://arxiv.org/pdf/2311.10533.pdf
This is the commit history: https://github.com/TkTech/can_ada/commits/main/ I guess you are right that there are 2 commits from a different dev, so it is technically not a solo project. I still wouldn't ever use this in production code.
...
can_ada is just the python bindings.
The actual underlying project is at https://github.com/ada-url/ada
Re: Parsing URLs in Python
#39Why not just use `httpx`? If you're not bound to the stdlib, it's a great alternative to `requests` and url parse
Re: Parsing URLs in Python
#40Hard to imagine the tradeoff of using a third party binary library developed this year vs just using urllib.parse being worth it. Is this solving a real problem?
According to itself, it's solving the issue of parsing differentials vulnerabilities: urllib.parse is ad-hoc and pretty crummy, and the headliner function "urlparse" is literally the one you should not use under any circumstance: it follows RFC 1808 (maybe, anyway) which was deprecated by RFC 2396 25 years ago . The odds that any other parser uses the same broken semantics are basically nil.
I agree that the stdlib parser is a mess, but as an observation: replacing one use of it with a (better!) implementation introduces a potential parser differential where one didn’t exist before. I’ve seen this issue crop up multiple times in real Python codebases, where a well-intentioned developer adds a differential by incrementally replacing the old, bad implementation.
That’s the perverse nature of “wrong but ubiquitous” parsers: unless you’re confident that your replacement is complete, you can make the situation worse, not better.