Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

eng.getwisdom.io

Post reply on HN
Print view View original

211–220 of 231 posts

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#211

Post by cryptonector »

ljm wrote: post #2

So if I understand this right, what GitHub did was something like: user = get_user_from_valid_email(params[:email]) send_reset_email(params[:email]) # instead of # send_reset_email(user.email) ? I've seen this pattern before and the reason is usually something about using the variable in memory as opposed to the function call. Total non-optimisation.

Or maybe someone wrote a validate_user_and_email() API and someone else wrote a send_password_reset_email() in a context where they lack access to the user DB, so they just validate the {username, email} and send to the attacker-provided email address.

The first case (yours) is a plain bug, while the latter (mine) is an architecture bug. Architecture bugs often arise out of organizational bugs.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#212

Post by elldoubleyew »

antris wrote: post #109
Avamander wrote: post #76

Earlier quoted context omitted.

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

Yes, all user-facing text should be Unicode, always. But ASCII has its use cases as well, as the parent comment mentioned in programming. I am natively non-ASCII compatible, but I am glad to program in ASCII and English. The only things that should not be English in written code are domain-specific terms that do not have an official unambiguous English translation. That's the only case to be made for non-ASCII charac…

>I think romanization can take care of it with most languages

Except for the ones that it really can't. When you try and romanize standard written Chinese it becomes nigh impossible to read for most speakers, often it would be more confusing than an English translation.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#213

Post by camgunz »

thaumasiotes wrote: post #208
camgunz wrote: post #201

Earlier quoted context omitted.

Wow what a resource, I love this. I'm a little skeptical about that table though, because as you point out, there's not a great way of figuring "meaning per character". For some more (real flawed) comparison, the English version of The Tale of Genji is ~60k words and 224 pages, and the Chinese version is ~75k words and 300 pages. [1] [2] So yeah, I'm skeptical about the claim that some languages have more meaning per…

> the Chinese version is ~75k words and 300 pages. [1] [2] Note that link [2] says "guess [of # of words] based on page count". 75k words, 300 pages is one statistic, not two separate statistics. (Estimating words based on page count is likely to be highly reliable, but still.) > I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall.…

Ugh, I was trying to get a good comparison of translations, but too bad I guess.

Yeah I mean, there's no question ideographic languages have more meaning per character than alphabetic languages. But other comparisons and considerations aren't as obvious:

- how do ideographic languages compare with each other?

- do people using ideographic languages write more?

- are ideographic languages as effective at compression as general (or special) compression algorithms?

Moving up the conceptual ladder from "average bytes per codepoint" to "average size of encoded tweet" (for example) is a big leap is all I'm saying.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#214

Post by morpheuskafka »

Symbiote wrote: post #136
_jal wrote: post #19

Earlier quoted context omitted.

> stripped of dots and other inert punctuation The period thing is a Gmail feature, not a standard. some.email@mydomain and someemail@mydomain most certainly do not deliver to the same mailbox.

It is actually an antifeature, since it is non-standard and leads to various attacks. For example, a malicious user can register some.email@gmail.com so.meemail@gmail.com someem.ail@gmail.com so.mee.mail@gmail.com somee.mail@gmail.com so..mee.mail@gmail.com som..eem.ail@gmail.com so.meemail@gmail.com somee.mail@gmail.com with a service, which will (quite reasonably) send a "Welcome" email. That results in a flood of…

I wouldn't go so far as to call it an antifeature--in fact, I wouldn't be surprised if a common use is to allow people to maintain multiple accounts on the same service with one email. It isn't standard, but it's not in violation of any standard--nothing says that the server must store each distinct valid email in a separate mailbox with its own login. Many servers implement "catchall" emails or aliases which result in the same thing, distinct addresses going to the same mailbox.

What is a problem, and what is non-standards-compliant, is GitHub incorrectly assuming that all mail providers will do this when many do not. It would be no different from assuming that "admin" and "postmaster" go the same mailbox because that's the way a lot of software is configured.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#215

Post by SAI_Peregrinus »

thaumasiotes wrote: post #142
jhanschoo wrote: post #101

Earlier quoted context omitted.

What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.

UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes

Yet the higher information density of Chinese & Japanese means the total file size tends to be about the same. Korean gets a bit short-changed, and some of the Indian subcontinent languages have low-density (like Latin) but high size (in with the Asian characters) and lose out.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#216

Post by nybble41 »

LeifCarrotson wrote: post #197
thaumasiotes wrote: post #157

Earlier quoted context omitted.

Hard to find, but it's the kind of thing that will hopefully come up in code review. Consider this pseudocode with helpful pseudo-hungarian notation: username = request.post_params('username') evil_email = request.post_params('email_address') user = get_user_by_name(username) good_email = user.email_address if good_email != evil_email: # Hackers! else: reset_password(user.id, evil_email) # it's fine; it's the same as…

The problem is that it requires a particularly detailed way of thinking about strings, much like the popular IEEE 754 floating point number question. If the code were instead written: username = request.post_params('username') email = request.post_params('email_address') user = get_user_by_name(username) if emails_are_equal(email, user.email_address) reset_password(user.id, email) Most people would not feel compelled…

> They may also have done things like normalize/remove optional.dots+suffixes@gmail.com, "quoted" or (commented) local parts.

At least the first two (removing dots and suffixes), along with conversion to lowercase, are strictly invalid transformations which may result in an email address referring to a different account. Sure, most email providers treat the account name as case-insensitive, and the use of '+' as a label/subaccount separator is a common convention, but neither of these is required. The RFCs say that the account name is case-sensitive (unlike the domain name) and '+' is just an ordinary part of the name; any special significance is assigned by the server. Ignoring '.' characters is something specific to Gmail.

In the context of a search it's not unreasonable to ignore some of these differences, but at that point the matching names aren't "equal", just "similar". Certainly it should never be assumed that it's safe to send email to the transformed version of the address.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#217

Post by thaumasiotes »

camgunz wrote: post #213
thaumasiotes wrote: post #208

Earlier quoted context omitted.

> the Chinese version is ~75k words and 300 pages. [1] [2] Note that link [2] says "guess [of # of words] based on page count". 75k words, 300 pages is one statistic, not two separate statistics. (Estimating words based on page count is likely to be highly reliable, but still.) > I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall.…

Ugh, I was trying to get a good comparison of translations, but too bad I guess. Yeah I mean, there's no question ideographic languages have more meaning per character than alphabetic languages. But other comparisons and considerations aren't as obvious: - how do ideographic languages compare with each other? - do people using ideographic languages write more? - are ideographic languages as effective at compression a…

> are ideographic languages as effective at compression as general (or special) compression algorithms?

That one we know; the compression algorithms are more compressive. For example, compressed Chinese text takes up less space than the same text uncompressed. Ideographic languages are still languages that real humans have to use, and they feature redundancy because that helps everyone. Compression algorithms have the luxury of stripping that redundancy out.

> how do ideographic languages compare with each other?

Of modern languages, only Chinese and Japanese could really be described as ideographic. (Japanese much more so than Chinese, in fact.) Chinese will have more meaning per character, because Japanese makes heavy use of the comparatively less meaningful kana. (Interestingly... it has to do this precisely because of its more ideographic nature.)

> do people using ideographic languages write more?

No idea.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#218

Post by merb »

BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

> but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area

nope. they are not byte arrays. if treating them as such, you run into many nasty bugs. today I encountered one (C#):

   if (string.Length > 60)
      string.Remove(60)
this has a fucking serious bug, especially if you try to show the string afterwards or worse save it inside a database.

strings are arrays of codepoints. NEVER TREAT THEM AS BYTE/CHAR ARRAYS.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#219

Post by BoppreH »

merb wrote: post #218
BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

> but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area nope. they are not byte arrays . if treating them as such, you run into many nasty bugs. today I encountered one (C#): if (string.Length > 60) string.Remove(60) this has a fucking serious bug, especially if you try to show the string afterw…

I meant "byte array" as in "this is a bunch of binary data you should not look into". Like people already do with images. You wouldn't remove a byte from the middle of a JPEG file, and you shouldn't do it with strings either.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#220

Post by 80386 »

gumby wrote: post #80
thewarpaint wrote: post #49

Earlier quoted context omitted.

Because it's part of the German language: https://en.m.wikipedia.org/wiki/Capital_%E1%BA%9E

Barely. If you look at it it is clearly just what its name says: a lowercase medial s combined with a lowercase z. The uppercase version exists, as a parallel commentator noted, basically as a typographic utility. Fraktur had/has other such lower case ligatures (tz, ch, sch, ss ( not ß) et al) but for some reason only ß survived into Latin script as a full fledged letter. I have a lot of old (mostly 20th century) boo…

Fraktur ß is a ligature of s and z; the current form came into use in Latin-script German because the Latin script already had ß, for the ligature of s and s.

Some sort of orthographic device is needed - s between two vowels means the first vowel is long and the consonant is voiced, and ss between two vowels means the first vowel is short and the consonant is voiceless. So it's useful to have a different case for when the first vowel is long and the consonant is voiceless:

Busen /bu:zən/

Busse /busə/

Buße /buːsə/

Vowel length isn't predictable from spelling in cases of consonant clusters and other digraphs: Hand /hant/ vs. Mond /moːnt/, Bruch /brux/ vs. Buch /buːx/, etc. So ss could've been used for both - or sz, although there are some words spelled with sz as a sequence of s and z, like Szene /stseːnə/.

Post reply on HN