Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

eng.getwisdom.io

Post reply on HN
Print view View original

201–210 of 231 posts

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#201

Post by camgunz »

hsivonen wrote: post #191
thaumasiotes wrote: post #181

Earlier quoted context omitted.

I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.

For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts

Wow what a resource, I love this.

I'm a little skeptical about that table though, because as you point out, there's not a great way of figuring "meaning per character". For some more (real flawed) comparison, the English version of The Tale of Genji is ~60k words and 224 pages, and the Chinese version is ~75k words and 300 pages. [1] [2]

So yeah, I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. I think it'd be cool to look at more data about this though, like (for example) stats from Treasure Data [3].

But regardless, it's pretty indisputable that UTF-16 is a lot better space-wise for languages that tend to be multibyte in UTF-8. Mostly what I'm trying (Quixotically) to say is "UTF-8 the world" ignores a lot of the world.

[1]: https://www.readinglength.com/book/isbn-4805314648

[2]: https://www.readinglength.com/book/isbn-7544717275

[3]: https://www.treasuredata.com/

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#202

Post by Keverw »

keyP wrote: post #151
Keverw wrote: post #99

Reminds me of a issue Spotify had too with unicodes and usernames. https://labs.spotify.com/2013/06/18/creative-usernames/ I had someone tell me that programming isn't real work before, and this is yet another example of all the small little details going into building things that most people don't really think about day to day. I haven't had to work with login code in a while, but might at some point. I know some sy…

> I had someone tell me that programming isn't real work before Haha, I don't even understand what metrics the person was using to consider something "work". A pilot sits throughout the flight but I think it's fair to say their working...

Yeah. I guess this person thinks sitting at a computer all day isn't real work. Real work would be working in a factory all your life breaking your back. Then I also think some people think making websites and coding is the same as using Word or Powerpoint. I guess they just don't really understand tech, probably a lot of people in the rust belt. Probably why they're driving young people away.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#203

Post by cdirkx »

simias wrote: post #129
Avamander wrote: post #76

Earlier quoted context omitted.

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

I'm very much in favor of UTF-8 everywhere, I think the pros outweigh the cons, but: >there are quite easy ways to avoid homograph attacks I'd like to hear about those because as far as I can tell it's still very much an unsolved problem.

The Unicode Technical Standard [1] (different from Unicode Standard) recommends treating identifiers (filenames, variable or fuction names, email adresses, usernames, etc.) different from normal text. There is a special class of 'identifier characters' which already exludes a lot of sneaky characters like invisible punctuation and obscure scripts that are not in modern use.

Additionally there are 5 additional restriction levels for identifiers depending on your specific situation:

1. ASCII only

2. single script

3. single script or Latin+{Japn, Hanb, Kore}

4. single script or Latin+{any, excluding Cyrillic, Greek}

5. Not containing any characters in the recommended blacklist of characters for use in secure contexts

6. No restrictions other than normal identifier restrictions

For any specific strings that might be confused, there is an algorithm to compute the visual 'skeleton' of a string and match it against that of another string to test if they are confusable.

[1] https://www.unicode.org/reports/tr39/

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#204

Post by paulddraper »

Freak_NL wrote: post #127
paulddraper wrote: post #122

Earlier quoted context omitted.

Though essentially every language has a romanization.

Often multiple, which makes the suggestion rather less suitable. Take a Japanese name like 麻生太郎. He may find his name romanized (ignoring the issue of surname/given name order) as Tarō Asō (preferable), Taro Aso (usually), Taroh Asoh (occurs on passports), Tarou Asou (‘I can't figure out diacritics’-style), Taroo Asoo, or even Tarô Asô.

Taro Aso would be the overwhelming choice for ASCII

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#205

Post by cdirkx »

squiggleblaz wrote: post #187
jcranmer wrote: post #184

Earlier quoted context omitted.

The simplest way to avoid most of them is to ban script mixing, especially mixing Latin, Greek, Cyrillic, and Cherokee.

So in a secure environment I can't write words in IPA? IPA requires mixing Greek beta and theta with Latin letters (but there's a Latin/IPA phi just for fun). It also doesn't help ɡ vs g. It sounds like any genuine solution is either not simple or excludes legitimate use cases.

Yes, the Unicode Technical standard identifies the Latin, Greek and Cyrillic (most of IPA characters) scripts as confusable, and recommends against mixing them in identifiers. Use in general text is fine though.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#206

Post by mrmcd »

evilotto wrote: post #113

That's the same glyph that got a guy killed. https://gizmodo.com/a-cellphones-missing-dot-kills-two-peopl...

My girlfriend is from Turkey, and I shared this story with her a while back (I discovered it while also researching some Unicode collision issues.) She said that while it's an entertaining story, there's almost certainly a bit of sensationalism and exaggeration from the Turkish press combined with credulity by English language journalists when re-reporting it. Basically, the supposed texts wouldn't have made sense in terms of grammar and syntax with a straightforward dotless/dotted I swap, and would've been obvious to someone fluent in Turkish what happened. This is would've been especially true if you had had this cell phone for any amount of time and communicating in Turkish and it had been routinely swapping Is.

More likely is a bunch of young and/or not too bright people were looking for a reason to get into a violent confrontation. Then the muckraking Turkish press had a sensationalist murder-suicide lovers quarrel story, and as a bonus a nationalistic "see how cell phone companies don't respect our culture" angle as the cherry on top.

However, you should still ALWAYS be careful when converting between character sets and be locality aware when manipulating strings. Practice string safety. ;)

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#207

Post by thaumasiotes »

LeifCarrotson wrote: post #197
thaumasiotes wrote: post #157

Earlier quoted context omitted.

Hard to find, but it's the kind of thing that will hopefully come up in code review. Consider this pseudocode with helpful pseudo-hungarian notation: username = request.post_params('username') evil_email = request.post_params('email_address') user = get_user_by_name(username) good_email = user.email_address if good_email != evil_email: # Hackers! else: reset_password(user.id, evil_email) # it's fine; it's the same as…

The problem is that it requires a particularly detailed way of thinking about strings, much like the popular IEEE 754 floating point number question. If the code were instead written: username = request.post_params('username') email = request.post_params('email_address') user = get_user_by_name(username) if emails_are_equal(email, user.email_address) reset_password(user.id, email) Most people would not feel compelled…

> There's a lot of magic - and a lot of risk - in an emails_are_equal method.

My whole point is that changing reset_password(user.id, email) to reset_password(user.id, user.email_address) neutralizes that risk. Then it doesn't matter whether emails_are_equal is risky or not, because a failure in emails_are_equal can only cause you to do a safe thing. What's less embarrassing -- "we accidentally sent a password reset for your account to you", or "we accidentally sent a password reset for your account to someone else"?

So you might not feel compelled to rewrite reset_password(user.id, email) as reset_password(user.id, user.email_address) -- but you should.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#208

Post by thaumasiotes »

camgunz wrote: post #201
hsivonen wrote: post #191

Earlier quoted context omitted.

For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts

Wow what a resource, I love this. I'm a little skeptical about that table though, because as you point out, there's not a great way of figuring "meaning per character". For some more (real flawed) comparison, the English version of The Tale of Genji is ~60k words and 224 pages, and the Chinese version is ~75k words and 300 pages. [1] [2] So yeah, I'm skeptical about the claim that some languages have more meaning per…

> the Chinese version is ~75k words and 300 pages. [1] [2]

Note that link [2] says "guess [of # of words] based on page count". 75k words, 300 pages is one statistic, not two separate statistics. (Estimating words based on page count is likely to be highly reliable, but still.)

> I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall.

Your skepticism is unwarranted. From a pure information-theoretic perspective, the claim that some languages have more meaning per character is a slam dunk, and less than a second of examination proves it conclusively.

For example, an entire English novel is unlikely to use more than 256 unique characters. A Chinese novel couldn't use anywhere near that few without sounding incredibly artificial.

Here are some single-character words in modern Mandarin:

    高 - high/tall
    低 - low
    大 - big
    小 - small
    最 - most (superlative marker)
    更 - more (comparative marker)
    到 - arrive
    走 - leave (go away)
    玩 - play (e.g. a game)
    看 - look
    听 - listen
    贵 - expensive
    爱 - love (verb)
    恨 - hate (verb)
    龍 - dragon
For something more representative, here are some song lyrics -- I'll enclose every word of multiple characters. Unenclosed stretches of text are words of one character each:

    都怪我不[小心]地
    [偷看了]你的[眼睛]
    [迷失了][自己]
    [不知][为什么][欢喜]
    [期待]听你的[声音]
    是[那么][甜蜜]
    
    我[拿着][电话]在你家门外
    想对你[说出]我[内心]中对你的[期待]
    [忽然][感觉][浑身][热血][澎湃]
    [今天]我[想要][[鼓起][勇气]][大声][表白]
How many distinct one-character words do you think English could practically support? How many twos? In this verse-and-a-half, there's one word that's always[1] three characters and two that have reached three characters by picking up a verb suffix, for a total of three words that are longer than two characters. (鼓起勇气 is kind of a special case, in that it's a well-known fixed expression, but its meaning is transparent as an ordinary combination of the two words 鼓起 and 勇气, which also see use outside the expression.)

[1] Actually, 什么 ["what"] has the vernacular contraction 啥, and this also applies to 为什么 ["why"], so you could argue that the word is sometimes just two characters.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#209

Post by londons_explore »

lazyjones wrote: post #164
ljm wrote: post #2

So if I understand this right, what GitHub did was something like: user = get_user_from_valid_email(params[:email]) send_reset_email(params[:email]) # instead of # send_reset_email(user.email) ? I've seen this pattern before and the reason is usually something about using the variable in memory as opposed to the function call. Total non-optimisation.

> Total non-optimisation. Lots of non-thought too. Sending e-mail directly from the place where web requests are processed isn't very smart. What if the SMTP subsystem is currently down or very slow? How many e-mails will you send if an attacker starts 1000 parallel web requests for a password reset? A saner way to do these things is to just set a flag ("password reset requested") in the user database there and do th…

Are designers like you the reason password reset mails frequently take 30 seconds plus to arrive?

I expect things on the web to be damn near instant - I want to click that password reset button and hear a synchronous "ding" of an arriving mail. I don't want to wait for some cron job to run once per minute.

If you want to send mail off the serving path, at least use a push based queue so there is no scheduling or polling involved.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#210

Post by thaumasiotes »

hsivonen wrote: post #191
thaumasiotes wrote: post #181

Earlier quoted context omitted.

I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.

For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts

> For writing globally appropriate software, UTF-8 is the best option that exists in practice.

OK, I lied. China doesn't use GB2312 anymore. They use its update, GB18030, which carries arbitrary Unicode data just as UTF-8 does... except that, like GB2312, it puts the Asian characters in two bytes and the European characters in three.

It's certainly not obvious to me that UTF-8 is better for globally appropriate software. It looks worse. Software for Europeans, sure.

Post reply on HN