Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

eng.getwisdom.io

Post reply on HN
Print view View original

91–100 of 231 posts

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#91

Post by BoppreH »

tim333 wrote: post #81
Iv wrote: post #59

Earlier quoted context omitted.

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and…

I was thinking that. Unicode is great for presenting text to users but for file names, email addresses, code and the like ASCII has a lot going for it.

It's not just Unicode weirdness, it's the whole concept of string operations. Examples:

1) Naive CSV libraries that break when given a field containing a comma itself. Same for injections.

2) Indexing things by strings (file names, user names, etc) relies on humans typing and reading with 100% accuracy.

3) Control characters are still present. Try to generate a file name containing every ASCII character (from 0x01 to 0x7F), then try to delete it in different ways. It's really frustrating.

4) Even string templates can cause problems. MMO's used to have game masters identified with "[GM] Character name", until scammers started using the same pattern. The sin here was to concatenate "[GM] " with the character name, which is spoofable, instead of using a badge or different color.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#92

Post by BoppreH »

camgunz wrote: post #89
BoppreH wrote: post #58

Earlier quoted context omitted.

The problem is now deeply entrenched, so I don't have perfect answers. But here are my guesses: > Who (process-wise) is responsible for converting bytes to pixels? The operating system, with minor exceptions (word processors, for example). Rendering logic is too complicated to be embedded into every application. And you get better accessibility and consistency. > How do users on social media put in their name? How is…

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point.

It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#93

Post by jcranmer »

iudqnolq wrote: post #65
deathanatos wrote: post #3

This blog sets opacity: 0 (fully invisible) on the entire content, then fails to unset that CSS with JS, b/c the JS crashes if you block cookies. > because the system lowercased the provided email address and compared it to the email address stored in the user database. While sending the email to the attack-provided email, instead of the one in the database, is bad… lowercasing emails is also not valid. The lookup sh…

A few tiny nitpicks: The local part MUST be treated as case-sensitive, servers are discouraged from doing so (as Gmail does): https://tools.ietf.org/html/rfc5321#section-2.4 Github shouldn't have normized case, but it's not insane to require lowercase in the first place I think so long as you don't convert silently. I wouldn't call +extensions and ignoring dots weird because Gmail does that and fairly or unfairly the…

Except that's not actually correct, because strictly following the ABNF does not yield correct semantics for an email address.

An email address consists of a local-part, a literal @ character, and then a domain name or an IP address literal. The local-part is either a series of dot-separated atoms (/[a-zA-Z0-9!#$%&'+/=?^_`{|}~-]+/ is the syntax for an atom) or a quoted string (/"([^\\"\0-\031\x7f]|\\[^\0-\031\x7f])"/ in regex). If you support EAI, you need to add \u00a0-\u10ffff [i.e., all non-ASCII non-C1 control characters].

According to RFC 822, you can insert whitespace (and comments) arbitrarily into the mailbox production, but that is non-semantic. Seeing "From: John Doe " means that the email address is exactly "foo@example.com", and you can reject any claim of the spelling without prejudicing any email addresses.

In practice, you can drop all support for quoted strings and IP address literals in most applications. So a correct email address (pre-EAI) regex in that vein would be:

    [a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*@([a-zA-Z0-9]+(-[a-zA-Z0-9]+)*\.)+[a-zA-Z]{2,}
If you insist on quoted-string support, then it looks like this:

    ([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*|"([^\\"\0-\031\x7f]|\\[\\"])*")@([a-zA-Z0-9]+(-[a-zA-Z0-9]+)*\.)+[a-zA-Z]{2,}
[I simplified the quoted string to only accept escapes that are semantically necessary--the strings "a b"@example.com and "a\ b"@example.com correspond to the same email address].

Edit: Sorry, there's quite a few asterisks in the regexes that Hacker News is turning into italicization, and I don't know how to unbork them.

Edit 2: Someone suggested how to unbork the standalone regexes, but the asterisks in the inline regex in the second paragraph are still missing.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#94

Post by BoppreH »

scythe wrote: post #82
BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

So here's the thing. When you display a word written entirely in Cyrillic characters, it would be wrong for it not to display as normal text. But when you display a word that contains eight Latin characters and one Cyrillic character, couldn't the display create some sort of warning? Highlight the section-mismatched character with a box? It's not normal in any language to mix alphabets.

That's a good heuristic, but you are still left with spoofing using 100% Cyrillic, and even old school approaches like "app1e.com" (1 instead of l). When you consider distracted users, or people with bad eyesight, you are back to square l.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#95

Post by iudqnolq »

fizx wrote: post #90
iudqnolq wrote: post #65

Earlier quoted context omitted.

A few tiny nitpicks: The local part MUST be treated as case-sensitive, servers are discouraged from doing so (as Gmail does): https://tools.ietf.org/html/rfc5321#section-2.4 Github shouldn't have normized case, but it's not insane to require lowercase in the first place I think so long as you don't convert silently. I wouldn't call +extensions and ignoring dots weird because Gmail does that and fairly or unfairly the…

That's more of a proof that regexes can't do everything, than a proof email is bad.

Pick a language of your choice, and fully implement the spec. I bet it'll still be long.

Edit: The following is completely wrong

For example, the python module to parse an email address is around 500 lines and repeatedly warns it'll be very hard to follow without a copy of the spec in front of you. It contains code for parsing multiple timezone formats and cite to a follow up spec addressing a bug in the initial treatment of negative timezones...

https://github.com/python/cpython/blob/master/Lib/email/_par...

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#96

Post by sophiebits »

hashStuff wrote: post #84

Wouldn't having the email hashed prevent this? Not sure why email addresses are stored plain text still. Especially if they are stored in databases.

Emails are used for, among other things, sending email to a user. Which requires the application know the email address.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#97

Post by iudqnolq »

jcranmer wrote: post #93
iudqnolq wrote: post #65

Earlier quoted context omitted.

A few tiny nitpicks: The local part MUST be treated as case-sensitive, servers are discouraged from doing so (as Gmail does): https://tools.ietf.org/html/rfc5321#section-2.4 Github shouldn't have normized case, but it's not insane to require lowercase in the first place I think so long as you don't convert silently. I wouldn't call +extensions and ignoring dots weird because Gmail does that and fairly or unfairly the…

Except that's not actually correct, because strictly following the ABNF does not yield correct semantics for an email address. An email address consists of a local-part, a literal @ character, and then a domain name or an IP address literal. The local-part is either a series of dot-separated atoms (/[a-zA-Z0-9!#$%&' +/=?^_`{|}~-]+/ is the syntax for an atom) or a quoted string (/"([^\\"\0-\031\x7f]|\\[^\0-\031\x7f])…

This is why I love HN. Someone replied to my joking improperly-researched comment with the right answer and a detailed explanation of why I'm wrong. I stand by my assertion that actually using that regex to check emails the user enters would be dumb.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#98

Post by BoppreH »

jkaptur wrote: post #86
BoppreH wrote: post #36

I love Unicode, but I'm more and more coming to the conclusion that strings are evil and should be treated as opaque byte arrays, whose only available operation is rendering into a bounded area. I now see any other string operation as code smell. It's scary how much of our infrastructure relies on strings, given how few guarantees string operations actually give. Take files names, for example. Two visually identical…

You’ve read my mind! I’ve been meaning to give a talk where I just go down the list of string methods and point out how each one is a hideous source of bugs.

Please, go forth and spread the word. Drop me an email (on my profile) if you want ideas.

I don't have a blog yet, but I already have a draft for a post titled "Strings are evil". There's a lot wrong with the current way we develop software, but I'm convinced this is a big one.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#99

Post by Keverw »

Reminds me of a issue Spotify had too with unicodes and usernames.

https://labs.spotify.com/2013/06/18/creative-usernames/

I had someone tell me that programming isn't real work before, and this is yet another example of all the small little details going into building things that most people don't really think about day to day.

I haven't had to work with login code in a while, but might at some point. I know some systems only allow alphanumeric usernames, but sounds like can't force people to have alphanumeric emails... Well I guess you could but might upset someone. I know there's normalizing functions though like NFD, NFC, NFKD or NFKC that might work for usernames but not sure what's really recommended.

Then also brute forcing attempts to try to mitigate attacks and other considerations to make also when building out an account system. Then if your company is large enough to provide phone support, not sure how you'd tell the support person which specific emoji you used in your username.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#100

Post by tzs »

microcolonel wrote: post #28
kevin_thibedeau wrote: post #27

Earlier quoted context omitted.

Senders don't get to dictate how a recipient encodes their addresses. RFC 822: The local-part of an addr-spec in a mailbox specification (i.e., the host's name for the mailbox) is understood to be whatever the receiving mail protocol server allows.

I am well aware of that, but I'm comfortable requiring that new customers don't register an account with an email address foolishly designed to resemble another customer's email address in this particular way. We don't throw away their specified mailbox address, we just don't accept registrations which look suspiciously similar, or intended to cause confusion. I repeat, this has absolutely nothing to do with the mail…

So if on my system we name users by their last name unless that is taken and then we add initials, and so Mike Nesmith (our only Nesmith) gets "nesmith", but we have several Smiths so Norman Edward Smith is given "n.e.smith", then only one of them can use your service?
Post reply on HN