Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

eng.getwisdom.io

Post reply on HN
Print view View original

101–110 of 231 posts

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#101

Post by jhanschoo »

camgunz wrote: post #89
BoppreH wrote: post #58

Earlier quoted context omitted.

The problem is now deeply entrenched, so I don't have perfect answers. But here are my guesses: > Who (process-wise) is responsible for converting bytes to pixels? The operating system, with minor exceptions (word processors, for example). Rendering logic is too complicated to be embedded into every application. And you get better accessibility and consistency. > How do users on social media put in their name? How is…

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

What's your reasoning?

Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#102

Post by ComputerGuru »

tim333 wrote: post #81
Iv wrote: post #59

Earlier quoted context omitted.

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and…

I was thinking that. Unicode is great for presenting text to users but for file names, email addresses, code and the like ASCII has a lot going for it.

No, it doesn’t. What’s a typical corporate email address? First.Last@...? F.Last@...?

More than half the world’s population does not use the Latin alphabet.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#103

Post by ComputerGuru »

scythe wrote: post #82
BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

So here's the thing. When you display a word written entirely in Cyrillic characters, it would be wrong for it not to display as normal text. But when you display a word that contains eight Latin characters and one Cyrillic character, couldn't the display create some sort of warning? Highlight the section-mismatched character with a box? It's not normal in any language to mix alphabets.

> it’s not normal in any language to mix alphabets

Decidedly not true. Languages like French and Turkish and German use their own alphabet but they look like the typical “American” Latin alphabet but aren’t. How do you decide if it’s English with non-American letters thrown in or not? Languages like Arabic are transliterated with English letters and Arabic numerals. Other Transliteration dialects use random Arabic characters with Latin others.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#104

Post by chupasaurus »

jcranmer wrote: post #93
iudqnolq wrote: post #65

Earlier quoted context omitted.

A few tiny nitpicks: The local part MUST be treated as case-sensitive, servers are discouraged from doing so (as Gmail does): https://tools.ietf.org/html/rfc5321#section-2.4 Github shouldn't have normized case, but it's not insane to require lowercase in the first place I think so long as you don't convert silently. I wouldn't call +extensions and ignoring dots weird because Gmail does that and fairly or unfairly the…

Except that's not actually correct, because strictly following the ABNF does not yield correct semantics for an email address. An email address consists of a local-part, a literal @ character, and then a domain name or an IP address literal. The local-part is either a series of dot-separated atoms (/[a-zA-Z0-9!#$%&' +/=?^_`{|}~-]+/ is the syntax for an atom) or a quoted string (/"([^\\"\0-\031\x7f]|\\[^\0-\031\x7f])…

IDNs (Internationalized Domain Names) would blow any regexp to an infinity because the list of languages usable in different domains isn't unified[0]

[0]https://www.iana.org/domains/idn-tables

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#105

Post by jcranmer »

iudqnolq wrote: post #95
fizx wrote: post #90

Earlier quoted context omitted.

That's more of a proof that regexes can't do everything, than a proof email is bad.

Pick a language of your choice, and fully implement the spec. I bet it'll still be long. Edit: The following is completely wrong For example, the python module to parse an email address is around 500 lines and repeatedly warns it'll be very hard to follow without a copy of the spec in front of you. It contains code for parsing multiple timezone formats and cite to a follow up spec addressing a bug in the initial trea…

The timezone isn't for parsing addressing headers, it's for parsing date headers. And actually, that file isn't for parsing email addresses, it's for parsing addressing headers in mail messages.

The code I wrote for parsing email headers is here: https://github.com/jcranmer/jsmime/blob/emailutils/headerpar... . A decent chunk of it is building a full lexer for email headers, and trying to cope with only supporting internationalization support in a few cases where they need to be supported. And the corner cases for that i18n support are really nasty.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#106

Post by ComputerGuru »

jcranmer wrote: post #93
iudqnolq wrote: post #65

Earlier quoted context omitted.

A few tiny nitpicks: The local part MUST be treated as case-sensitive, servers are discouraged from doing so (as Gmail does): https://tools.ietf.org/html/rfc5321#section-2.4 Github shouldn't have normized case, but it's not insane to require lowercase in the first place I think so long as you don't convert silently. I wouldn't call +extensions and ignoring dots weird because Gmail does that and fairly or unfairly the…

Except that's not actually correct, because strictly following the ABNF does not yield correct semantics for an email address. An email address consists of a local-part, a literal @ character, and then a domain name or an IP address literal. The local-part is either a series of dot-separated atoms (/[a-zA-Z0-9!#$%&' +/=?^_`{|}~-]+/ is the syntax for an atom) or a quoted string (/"([^\\"\0-\031\x7f]|\\[^\0-\031\x7f])…

> Edit: Sorry, there's quite a few asterisks in the regexes that Hacker News is turning into italicization, and I don't know how to unbork them.

Prefix the regex with four spaces.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#107

Post by inimino »

Domains are not case sensitive, but email local parts are! There is no reason whatsoever to do case normalization on local parts of emails on any domain you do not own, as this is strictly incorrect and could lead to a totally different address that also exists (as happened here).

Of course, email providers are free to do whatever case folding or normalization they want, in which case the security burden of avoiding collisions is on the provider. If someone's email provider maps different case variants to the same mailbox, there's still no need whatsoever to do anything to the address, as the user will get it delivered to them regardless. If the provider doesn't do case folding, they will have to enter their local part case sensitively, but that's exactly the same as for any other use of their email address.

I can only imagine how this vulnerability came to be. Unicode is not to blame here. If security-critical password reset code was not audited carefully enough to catch a mistake like this, one wonders what other errors might remain.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#108

Post by jcranmer »

chupasaurus wrote: post #104
jcranmer wrote: post #93

Earlier quoted context omitted.

Except that's not actually correct, because strictly following the ABNF does not yield correct semantics for an email address. An email address consists of a local-part, a literal @ character, and then a domain name or an IP address literal. The local-part is either a series of dot-separated atoms (/[a-zA-Z0-9!#$%&' +/=?^_`{|}~-]+/ is the syntax for an atom) or a quoted string (/"([^\\"\0-\031\x7f]|\\[^\0-\031\x7f])…

IDNs (Internationalized Domain Names) would blow any regexp to an infinity because the list of languages usable in different domains isn't unified[0] [0] https://www.iana.org/domains/idn-tables

By the time you hit IDNs, regexes for validation are no longer your biggest issue. Your real check for validity at that point becomes "can I actually contact the host" (or send an email, if validating an email address), and there is little point in aggressively validating a purported domain name instead of checking if it actually exists.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#109

Post by antris »

Avamander wrote: post #76
Iv wrote: post #59

Earlier quoted context omitted.

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and…

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII characters in programming that I can think of, but I think romanization can take care of it with most languages.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#110

Post by rgj »

BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

Strings are not evil - user supplied input should be treated as evil.

The fact that most user input is supplied as a string is a mere coincidence.

Post reply on HN