Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' • Hacker News Board

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#121

Post by lopsidedBrain » Tue, Dec 17, 2019, 7:23 AM UTC

camgunz wrote: post #89

BoppreH wrote: post #58

Earlier quoted context omitted.

The problem is now deeply entrenched, so I don't have perfect answers. But here are my guesses: > Who (process-wise) is responsible for converting bytes to pixels? The operating system, with minor exceptions (word processors, for example). Rendering logic is too complicated to be embedded into every application. And you get better accessibility and consistency. > How do users on social media put in their name? How is…

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. webkit-internals and UI rendering frameworks). Okay, maybe they are also kind of useful for sorting and collating. But fixed-width character encodings are almost never useful, and invite programmer to make assumptions about how strings can be sliced.

Most server-side applications should never have to know what these concepts even are. Or any library that is not user-facing. They get bytes from the UI layer, and they can keep them as opaque bytes. For user-facing apps, you can ask your renderer library for a pixel-width or similar for a string, and let them handle how to parse it. Very little code ever needs to know about unicode.

Any kind of input-sanitization is vastly simplified in utf8, and that makes it worth it for me. For me the really troubling trends are conventions like Rust Utf8Error, where they can cause what I'd consider a UI-related exception in code that had no business even interpreting what those bytes are. Unfortunately, every API uses strings, so they are kind of hard to avoid. It introduces what I'd consider a software layering problem.

Maybe others here with more experience with internationalization can chime in and tell me I'm wrong.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#122

Post by paulddraper » Tue, Dec 17, 2019, 7:27 AM UTC

ComputerGuru wrote: post #102

tim333 wrote: post #81

Earlier quoted context omitted.

I was thinking that. Unicode is great for presenting text to users but for file names, email addresses, code and the like ASCII has a lot going for it.

No, it doesn’t. What’s a typical corporate email address? First.Last@...? F.Last@...? More than half the world’s population does not use the Latin alphabet.

Though essentially every language has a romanization.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#123

Post by paulddraper » Tue, Dec 17, 2019, 7:32 AM UTC

lonelappde wrote: post #72

iudqnolq wrote: post #65

Earlier quoted context omitted.

A few tiny nitpicks: The local part MUST be treated as case-sensitive, servers are discouraged from doing so (as Gmail does): https://tools.ietf.org/html/rfc5321#section-2.4 Github shouldn't have normized case, but it's not insane to require lowercase in the first place I think so long as you don't convert silently. I wouldn't call +extensions and ignoring dots weird because Gmail does that and fairly or unfairly the…

Please don't pollute the HN UI with garbage

Point proven I guess.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#124

Post by ksec » Tue, Dec 17, 2019, 7:34 AM UTC

antris wrote: post #109

Avamander wrote: post #76

Earlier quoted context omitted.

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII charac…

This, I think people are arguing for ASCII in programming, not going back to ASCII in general.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#125

Post by oaiey » Tue, Dec 17, 2019, 7:59 AM UTC

ljm wrote: post #2

So if I understand this right, what GitHub did was something like: user = get_user_from_valid_email(params[:email]) send_reset_email(params[:email]) # instead of # send_reset_email(user.email) ? I've seen this pattern before and the reason is usually something about using the variable in memory as opposed to the function call. Total non-optimisation.

I always preferred using the record instead of the user input for follow up operations like that.

So far it was only a gut feeling. Now I know why I do it ;)

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#126

Post by ishi » Tue, Dec 17, 2019, 8:02 AM UTC

blipmusic wrote: post #115

Julia returns: julia> c='ı' 'ı': Unicode U+0131 (category Ll: Letter, lowercase) julia> uppercase(c) 'I': ASCII/Unicode U+0049 (category Lu: Letter, uppercase) julia> lowercase(uppercase(c)) 'i': ASCII/Unicode U+0069 (category Ll: Letter, lowercase) Is this something that needs changing in the Unicode spec itself or how strings are handled in general by various tools/programming languages? I love [plain] text, but it…

I believe this specific problem exists in Unicode, and not just in the Julia language.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#127

Post by Freak_NL » Tue, Dec 17, 2019, 8:02 AM UTC

paulddraper wrote: post #122

ComputerGuru wrote: post #102

Earlier quoted context omitted.

No, it doesn’t. What’s a typical corporate email address? First.Last@...? F.Last@...? More than half the world’s population does not use the Latin alphabet.

Though essentially every language has a romanization.

Often multiple, which makes the suggestion rather less suitable.

Take a Japanese name like 麻生太郎. He may find his name romanized (ignoring the issue of surname/given name order) as Tarō Asō (preferable), Taro Aso (usually), Taroh Asoh (occurs on passports), Tarou Asou (‘I can't figure out diacritics’-style), Taroo Asoo, or even Tarô Asô.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#128

Post by Freak_NL » Tue, Dec 17, 2019, 8:09 AM UTC

lopsidedBrain wrote: post #121

camgunz wrote: post #89

Earlier quoted context omitted.

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. web…

Also, UTF-16 is not fixed width in modern usage. As soon as someone uses an emoji, boom!, surrogate pairs. So unless UTF-32 is used at a glorious four bytes per character, you won't get fixed width.

> For me the really troubling trends are conventions like Rust Utf8Error,[…]

Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? How is Rust's approach different from other languages?

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#129

Post by simias » Tue, Dec 17, 2019, 8:24 AM UTC

Avamander wrote: post #76

Iv wrote: post #59

Earlier quoted context omitted.

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and…

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

I'm very much in favor of UTF-8 everywhere, I think the pros outweigh the cons, but:

>there are quite easy ways to avoid homograph attacks

I'd like to hear about those because as far as I can tell it's still very much an unsolved problem.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#130

Post by simias » Tue, Dec 17, 2019, 8:30 AM UTC

BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

The problem is that your method works well if the string is either:

- Only meant to be used in a machine-to-machine interface (like a JSON key for instance).

- Only meant to be used in a machine-to-human interface (like the text letting you know that you used a wrong password).

For strings that have a special significance to both humans and machines you can still run into problems, mainly because of the many unicode strings that look similar (or sometimes even identical) visually but are encoded with a different byte sequence. Take usernames or URLs for instance.

But in the end Unicode is messy because human languages are messy. As programmers we like well-bounded problems with elegant generic solutions. This clearly isn't possible here and we have to deal with it.