71–80 of 99 posts
Hard to imagine the tradeoff of using a third party binary library developed this year vs just using urllib.parse being worth it. Is this solving a real problem?
Writing a new parser in C++ is a mistake IMO. At the very least, you need to write a fuzzer. At best, you should be using one of the many memory safe languages available to you. I retract my criticism if this project is just for fun. Edit: downvoters, do you disagree? Edit2: OK, I may have judged a bit prematurely. Ada itself has fuzzers and tests. They're just not exported to the can_ada project.
Earlier quoted context omitted.
Because that's the Punycode representation: https://en.wikipedia.org/wiki/Punycode https://www.punycoder.com/
I wasn't aware of this, I'd seen those URLs before but only in the context of Chinese ones and thought it was Chinese-specific. It's interesting because I just went down an apparent rabbit hole inplementing Byte-level encoding for using language models with unicode. There each byte in a unicode character is mapped to a printable character that goes up to 255 See https://github.com/openai/gpt-2/blob/9b63575ef42771a015…
Okay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.
I was wondering, can that clash with a "normal" domain registered as "xn--....."? Apparently there is another specific rule in RFC 5891 saying "The Unicode string MUST NOT contain "--" (two consecutive hyphens) in the third and fourth character positions" [0]
Also, if I was forced to represent Unicode as ASCII, punycode encoding is not the obvious one - it's pretty confusing. But, I don't know much about how and why it was chosen, so I assume there's good reason.
[0] https://datatracker.ietf.org/doc/html/rfc5891#section-4.2.3....
Earlier quoted context omitted.
It seems unlikely that this C++ library written by a solo dev is somehow more secure than the Python standard library would be for such a security-sensitive task.
Hi, can_ada (but not ada!) dev here. Ada is over 20k lines of well-tested and fuzzed source by 25+ developers, along with an accompanying research paper. It is the parser used in node.js and parses billions of URLs a day. can_ada is simply a 60-line glue and packaging making it available with low overhead to Python.
Despite my snarky comments, thank you for contributing to the python ecosystem, this does seem like a cool project for high performance URL parsing!
Writing a new parser in C++ is a mistake IMO. At the very least, you need to write a fuzzer. At best, you should be using one of the many memory safe languages available to you. I retract my criticism if this project is just for fun. Edit: downvoters, do you disagree? Edit2: OK, I may have judged a bit prematurely. Ada itself has fuzzers and tests. They're just not exported to the can_ada project.
Ada developer here. Ada has more than 5000 tests, is included in oss-fuzz project and battle tested in Node.js and Cloudflare workers.
I didn't understand that can_ada is not where the parser is developed.
This is intriguing to me for the performance and correctness reasons, but also if it makes the result more dev friendly than the urllib.parse tiple-ish-object result thing.
I posted this elsewhere in the thread, but there absolutely is prior art here. Check out Yarl (urllib.parse wrapper with the nicer interface) and Hyperlink (green field, immutable OO style, focus on correctness). Both on PyPI for many years now.
No mention of prior art? The Hyperlink library has stated correctness as its goal for a long time: https://pypi.org/project/hyperlink Of course there is always room for new projects, but it still feels weird to act as if this is the first time anybody has ever tried this. It seems like a lot of people are under this same mistaken impression, at least according to the sample of HN users who commented in this thread.
Okay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.
Okay so some googling found me that the "xn--" means the rest of the hostname will be unicode, but why does é become -fsa in www.xn--googl-fsa.com. Google failed on the second part.
This system seems pretty weird to me. I was wondering, can that clash with a "normal" domain registered as "xn--....."? Apparently there is another specific rule in RFC 5891 saying "The Unicode string MUST NOT contain "--" (two consecutive hyphens) in the third and fourth character positions" [0] Also, if I was forced to represent Unicode as ASCII, punycode encoding is not the obvious one - it's pretty confusing. But…