Hacking GitHub's Auth with Unicode's Turkish Dotless 'I' • Hacker News Board

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#181

Post by thaumasiotes » Tue, Dec 17, 2019, 2:54 PM UTC

jhanschoo wrote: post #170

thaumasiotes wrote: post #142

Earlier quoted context omitted.

UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes

This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.

I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#182

Post by sfilipov » Tue, Dec 17, 2019, 2:55 PM UTC

squiggleblaz wrote: post #177

pjc50 wrote: post #174

Earlier quoted context omitted.

This is one of those cases where what we want is a tainting system for strings, not an encoding problem. The only language I've seen attempt this was Perl, and even then intermittently. It should be made as difficult as possible to pass user input directly to something vulnerable like an email-sending API, without first laundering it through "validation". Unfortunately it can be very hard to do good validation, but i…

You could do that with a modern type system. For instance, you create types: UserInputString, UnvalidatedEmailString, ValidatedEmailAddress. At least the first two of them are type wrappers for strings - opaque to the type checker, but transparent to the runtime. Rust and Haskell have wrappers like this. Typescript does not. Keep the internal details private to the module so that application code doesn't concern itse…

> At least the first two of them are type wrappers for strings - opaque to the type checker, but transparent to the runtime. Rust and Haskell have wrappers like this. Typescript does not.

You are right that TypeScript aliases don't work like Haskell ones (which are considered different, and type checked). In TypeScript you can use "branded types" to work around the more loose structural typing:

  type Firstname = string & { readonly brand?: unique symbol }

https://stackoverflow.com/a/51578232

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#183

Post by jmkni » Tue, Dec 17, 2019, 2:58 PM UTC

thaumasiotes wrote: post #149

BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

I want to note a separate issue of defensive coding that comes up in the writeup: > GitHub's forgot password feature could be compromised because the system lowercased the provided email address and compared it to the email address stored in the user database. If there was a match, GitHub would send the reset password link to the email address provided by the attacker The logical flow is: 1. Get the email address fro…

There must be so many websites vulnerable to this sort of attack, or similar.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#184

Post by jcranmer » Tue, Dec 17, 2019, 3:41 PM UTC

simias wrote: post #129

Avamander wrote: post #76

Earlier quoted context omitted.

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

I'm very much in favor of UTF-8 everywhere, I think the pros outweigh the cons, but: >there are quite easy ways to avoid homograph attacks I'd like to hear about those because as far as I can tell it's still very much an unsolved problem.

The simplest way to avoid most of them is to ban script mixing, especially mixing Latin, Greek, Cyrillic, and Cherokee.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#185

Post by squiggleblaz » Tue, Dec 17, 2019, 3:42 PM UTC

ksec wrote: post #124

antris wrote: post #109

Earlier quoted context omitted.

Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII charac…

This, I think people are arguing for ASCII in programming, not going back to ASCII in general.

But aside from APL and a few Haskell coders, who isn't programming in ascii? I mean, sometimes i chuck an emoji into a comment but I don't think that really counts.

And how could programming in ascii have solved this bug? The problem is that the strings were compared without giving any thought to what "are these strings equal" is supposed to mean.

The only general solution to this kind of bug - that is, to considering distinct emails to be identical - is to have a special function that can check if two emails are identical according to the spec. This function will be different than determining if two names are the same in an American database where dotless and dotful i are the same, and that function will also be different than a function determining if two names are the same in a Turkish database where dotless and dotful i are different. Although I don't know why you're trying to compare strings for identity, you're probably just after similarity anyway - how often do we convert strings to lowercase before we compare them.

Generic string equality functions are always the source of bugs, since there's no useful, single uniformally applicable definition of string equality.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#186

Post by rsync » Tue, Dec 17, 2019, 3:43 PM UTC

Avamander wrote: post #76

Iv wrote: post #59

Earlier quoted context omitted.

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and…

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

"I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages."

You are correct that most of the world is non-ascii and I am enthusiastic about recognizing that diversity and I am willing to pay certain costs in return for the richness it provides.

However, I will point out that certain systems have been deemed crucial and in need of deliberate (and brutal) dumbing-down. Specifically, I speak of the global Air Traffic Control system that is English only[1].

Tagalog/Flemish/Satsugu is hard. Let's (land airplanes with) English.

[1] https://en.wikipedia.org/wiki/Aviation_English

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#187

Post by squiggleblaz » Tue, Dec 17, 2019, 3:48 PM UTC

jcranmer wrote: post #184

simias wrote: post #129

Earlier quoted context omitted.

I'm very much in favor of UTF-8 everywhere, I think the pros outweigh the cons, but: >there are quite easy ways to avoid homograph attacks I'd like to hear about those because as far as I can tell it's still very much an unsolved problem.

The simplest way to avoid most of them is to ban script mixing, especially mixing Latin, Greek, Cyrillic, and Cherokee.

So in a secure environment I can't write words in IPA? IPA requires mixing Greek beta and theta with Latin letters (but there's a Latin/IPA phi just for fun).

It also doesn't help ɡ vs g.

It sounds like any genuine solution is either not simple or excludes legitimate use cases.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#188

Post by codetrotter » Tue, Dec 17, 2019, 3:49 PM UTC

eyegor wrote: post #176

jazzabeanie wrote: post #161

Earlier quoted context omitted.

If they do have multiple email addresses associated with one account, and they don't want to send a password reset to all of them, then you can see how it would happen. ``` if evil_email not in good_email_addresses: # Hackers! else: # just reset with provided email. If I thought there was a potential security issue I would have already addressed it. ``` So easy to be lazy at this point, especially under time pressure…

Off topic, you need the triple backticks on their own lines for them to render here.

Sorry but that is not correct either. Code blocks on HN are created by using two or more leading spaces on each line.

Tripple backticks do nothing, and will be shown on the same line as the next line even if put on a separate line, because they are considered as regular text and part of one paragraph of text.

See https://news.ycombinator.com/formatdoc

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#189

Post by jcranmer » Tue, Dec 17, 2019, 3:54 PM UTC

jiggawatts wrote: post #165

atoav wrote: post #145

Earlier quoted context omitted.

The longer I program, the more I am convinced that falling back to ASCII in 90% of (non-embedded) use-cases is just a programmer avoiding to do the extra brainwork to deal with encodings. That is why I like the Rust approach: make these issues front and center, implement clear solutions for common use cases and enforce them. In to many languages encodings feel like an afterthought rather than something that has been…

The issue I see with rust is that they (partially) repeated the mistake older operating systems made: they assumed that the Unicode of that time will be Unicode forever. While UTF8 is just an encoding, a Rust UTF8 "String" is actually a UTF8-encoded Unicode 11.0 string. Or version 12.0, possibly 12.1. Maybe 13 soon! Who knows. A moving target, certainly. If you're "agile", you can just recompile and you will be fine,…

The Unicode consortium has guaranteed that there are many properties they won't change once they're released. Even if the property value is objectively wrong.

More to the point: the standard library of Rust relies on no properties of Unicode that will change--there's no builtin normalization, case folding, grapheme cluster support, etc. So there's no data tables in the standard library that need to be updated. The only assumption Rust makes of Unicode is that no assigned codepoints will be changed, and that the space will not grow beyond the current limit of U+10FFFF.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#190

Post by bregma » Tue, Dec 17, 2019, 3:55 PM UTC

knolax wrote: post #150

Avamander wrote: post #76

Earlier quoted context omitted.

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

I swear software's biggest problem is that developers prioritize their own ergonomics over actually producing functional code. Imagine if auto engineers just decided "safety is hard, let's just make deathtraps". I know the current narrative is that the 737 Max failed because of MBAs but it's a really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that cause…

It was a bad corporate ecosystem that caused the crash. The people at fault were the accountants that tweaked lines un Excel until they got the numbers they wanted amd then pressed reality into the service of those numbers. Bad software didn't eliminate pilot training on the new systems. Bad software didn't shortcut the required FAA safety certification on the system. Bad software didn't reduce the number of redundant sensors that fed the system.

Bad beancounters did all that. C-levels in suits who got million-dollar bonuses for killing 346 people to gain marginally increased quarterly results. Don't blame the software.