Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

eng.getwisdom.io

Post reply on HN
Print view View original

141–150 of 231 posts

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#141

Post by atoav »

Freak_NL wrote: post #128
lopsidedBrain wrote: post #121

Earlier quoted context omitted.

I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. web…

Also, UTF-16 is not fixed width in modern usage. As soon as someone uses an emoji, boom! , surrogate pairs. So unless UTF-32 is used at a glorious four bytes per character, you won't get fixed width. > For me the really troubling trends are conventions like Rust Utf8Error,[…] Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? How is Rust's approach different from other langu…

> Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence?

As a Rust user: this is what your function returns if the user inputs non-UTF-8 bytes into something that expects UTF-8 bytes and the programmer explicitly choses not to handle that error.

I don’t see anything wrong with that. Sometimes you might be interested in receiving, processing, storing valid UTF-8 strings rather than arbitrary byte sequences, that may or may not be able to be translated back into something valid that you can display.

I always hated to do encoding related work with a passion before I started using Rust. Rust forced me to do it the right way and actually understand why I am doing it a certain way. When it comes to encoding I feel safer in Rust then e.g. in Python, despite having used it for four times as long.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#142

Post by thaumasiotes »

jhanschoo wrote: post #101
camgunz wrote: post #89

Earlier quoted context omitted.

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.

UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system:

    ASCII:                  1 byte
    European characters:    2 bytes
    Asian characters:       3 bytes
There is no reason you'd want this in Asia. China uses GB2312, which flips it around:

    ASCII:                  1 byte
    Asian characters:       2 bytes
    European characters:    3 bytes

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#143

Post by stedaniels »

_jg wrote: post #119
mrighele wrote: post #114

> // Note the Turkish dotless i 'John@Gıthub.com'.toLowerCase() === 'John@Github.com'.toLowerCase() I'm not sure this example is correct. The dotless ı is already lower cased, so the comparison above should yield false. Maybe the author was thinking about upper case dotted "İ", which becomes regular dotted "i" when lower cased. So what could happen is that an user enter "JOHN@GİTHUB.COM" as email, and then the email…

Should be: 'John@Gıthub.com'.toUpperCase() === 'John@Github.com'.toUpperCase()

The example doesn't seem to work at all for GitHub's explanation.. they say that their outgoing email server didn't support unicode in the domain part anyway. What am I missing? Was your actual attack on the local part?

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#145

Post by atoav »

Iv wrote: post #59
BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and…

The longer I program, the more I am convinced that falling back to ASCII in 90% of (non-embedded) use-cases is just a programmer avoiding to do the extra brainwork to deal with encodings.

That is why I like the Rust approach: make these issues front and center, implement clear solutions for common use cases and enforce them. In to many languages encodings feel like an afterthought rather than something that has been considered from the start. When I started using Rust I e.g. learned that the Strings a OS uses in its filenames are not necessarily valid UTF-8. Rust forces you to handle this explicitly. Strings are complex and hiding this can be dangerous

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#146

Post by thaumasiotes »

qwerty456127 wrote: post #137

> 'ß'.toLowerCase() // 'ss' Why? First of all ß is already lowercase, why should toLowerCase() change it? It also is a normal letter having both uppercase (ẞ) and lowercase (ß) forms so converting between the cases can be made be trivial and quirk-free. Arguably the most common word you will encounter ß in is Straße (street) where it already is lowercase - will "Straße".toLowerCase() turn it into "strasse"? WTF? "Str…

> (nevertheless "Straße".upper() actually returns "STRASSE" ignoring the existence of the uppercase ẞ (U+1E9E))

Python probably predates the addition of a notional capital ß glyph in 2017. SS is the capitalization of ß that you'd expect if you were thinking of your data as a string rather than a collection of font elements.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#147

Post by qwerty456127 »

iraldir wrote: post #135
iforgotpassword wrote: post #133

Earlier quoted context omitted.

And how's that supposed to work? Every country has developed some form of transliteration to the Latin alphabet. Its simple enough that anybody can memorize these additional 26 letters. A form of least common denominator. Anyone on this planet can scribble down their email address on a piece of paper and equally type one on their computer. Now imagine you'd write down a Chinese email address with characters. How am I…

Counterpoint: A lot of people in China are as familiar with the latin alphabet as you are with the Cyrillic alphabet: Source: https://newrepublic.com/article/117608/chinese-number-websit...

I've once seen a Chinese girl texting on a bus stop. She used the ordinary US-looking on-screen keyboard but Chinese hieroglyphs were appearing in the message window as she typed.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#148

Post by mnw21cam »

founderling wrote: post #138
BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

opaque byte arrays, only available operation is rendering into a bounded area Goodbye web then. Because the whole request you get from a client is nothing but strings. So when you run an onlineshop and a customer orders "7" screwdrivers - then you are screwed. Because what does an "opaque byte array" of "opaque byte arrays" cost? How much shipping will that be? But at least you have a brand new customer: Henry@gmail.…

> But at least you have a brand new customer: Henry@gmail.com. Since you do not lowercase the email and do not recognize him as henry@gmail.com which he used in his last order.

Which would be correct behaviour. From RFC 5321, part 2.4, "The local-part of a mailbox MUST BE treated as case sensitive."

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#149

Post by thaumasiotes »

BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

I want to note a separate issue of defensive coding that comes up in the writeup:

> GitHub's forgot password feature could be compromised because the system lowercased the provided email address and compared it to the email address stored in the user database. If there was a match, GitHub would send the reset password link to the email address provided by the attacker

The logical flow is:

1. Get the email address from the forgot-password request.

2. Get the email address from the database for the same account.

3. Check whether they match.

4a. If not, we're under attack -- refuse the request.

4b. If so, all is well -- send a password reset to the email address.

Of course, we know the email address twice -- we asked the user for it during the password reset process (step 1), but we never needed to do that because we already had an email address on file for the account. We retrieved that email address in step 2. We know that the two addresses are the same, but, if you look at the semantics behind the variables, in step 4b we're choosing one of these two "equivalent" options, depending on which variable we use for the email address:

1. Send the account password to the account owner.

2. Send the account password to a guy who doesn't know what the password is.

And these have very different risk profiles. Choosing the first option instead of the second would have prevented this attack without needing to worry about unicode case-translation issues. You never want to trust information you just received from an unknown user when you already have the same information from a more authoritative source.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#150

Post by knolax »

Avamander wrote: post #76
Iv wrote: post #59

Earlier quoted context omitted.

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and…

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

I swear software's biggest problem is that developers prioritize their own ergonomics over actually producing functional code. Imagine if auto engineers just decided "safety is hard, let's just make deathtraps". I know the current narrative is that the 737 Max failed because of MBAs but it's a really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that caused the crash.
Post reply on HN