Why not just use `httpx`? If you're not bound to the stdlib, it's a great alternative to `requests` and url parse
[deleted]
Parsing URLs in Python
41–50 of 99 posts
Re: Parsing URLs in Python
#42The Ada programming language is cursed with overlapping acronyms. GPS, ADA, SPARK, AWS. Seems it just got a little bit worse.
> Ada is a WHATWG-compliant and fast URL parser written in modern C++ Why would you do that, Daniel?
Hah :) It's named after Yagiz Nizipli's (Ada dev) newborn daughter, Ada.
Re: Parsing URLs in Python
#43Earlier quoted context omitted.
According to itself, it's solving the issue of parsing differentials vulnerabilities: urllib.parse is ad-hoc and pretty crummy, and the headliner function "urlparse" is literally the one you should not use under any circumstance: it follows RFC 1808 (maybe, anyway) which was deprecated by RFC 2396 25 years ago . The odds that any other parser uses the same broken semantics are basically nil.
It seems unlikely that this C++ library written by a solo dev is somehow more secure than the Python standard library would be for such a security-sensitive task.
Hi, can_ada (but not ada!) dev here. Ada is over 20k lines of well-tested and fuzzed source by 25+ developers, along with an accompanying research paper. It is the parser used in node.js and parses billions of URLs a day.
can_ada is simply a 60-line glue and packaging making it available with low overhead to Python.
Re: Parsing URLs in Python
#44Looking at the github site for can_ada, I discovered that the developers live in Montreal, Canada. Nice one.
Re: Parsing URLs in Python
#45[dead]
Re: Parsing URLs in Python
#46Re: Parsing URLs in Python
#47Looking at the github site for can_ada, I discovered that the developers live in Montreal, Canada. Nice one.
[deleted]
Re: Parsing URLs in Python
#48A great initiative!
We need a better URL parser in Scrapy, for similar reasons. Speed and WHATWG standard compliance (i.e. do the same as web browsers) are the main things.
It's possible to get closer to WHATWG behavior by using urllib and some hacks. This is what https://github.com/scrapy/w3lib does, which Scrapy currently uses. But it's still not quite compliant.
Also, surprisingly, on some crawls URL parsing can take CPU amounts similar to HTML parsing.
Ada / can_ada look very promising!
Re: Parsing URLs in Python
#49A great initiative! We need a better URL parser in Scrapy, for similar reasons. Speed and WHATWG standard compliance (i.e. do the same as web browsers) are the main things. It's possible to get closer to WHATWG behavior by using urllib and some hacks. This is what https://github.com/scrapy/w3lib does, which Scrapy currently uses. But it's still not quite compliant. Also, surprisingly, on some crawls URL parsing can t…
can_ada dev here. Scrapy is a fantastic project, we used it extensively at 360pi (now Numerator), making trillions of requests. Let me know if I can help :)
Re: Parsing URLs in Python
#50Earlier quoted context omitted.
According to itself, it's solving the issue of parsing differentials vulnerabilities: urllib.parse is ad-hoc and pretty crummy, and the headliner function "urlparse" is literally the one you should not use under any circumstance: it follows RFC 1808 (maybe, anyway) which was deprecated by RFC 2396 25 years ago . The odds that any other parser uses the same broken semantics are basically nil.
I agree that the stdlib parser is a mess, but as an observation: replacing one use of it with a (better!) implementation introduces a potential parser differential where one didn’t exist before. I’ve seen this issue crop up multiple times in real Python codebases, where a well-intentioned developer adds a differential by incrementally replacing the old, bad implementation. That’s the perverse nature of “wrong but ubi…
> unless you’re confident that your replacement is complete
And that any 3rd party libs you use also don't ever call the stdlib parser internally because you do not want to debug why a URL works through some code paths but not others.
Turns out that url parsing is a cross-cutting concern like logging where libs should defer to the calling code's implementation but the Python devs couldn't have known that when this module was written.