Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

eng.getwisdom.io

Post reply on HN
Print view View original

191–200 of 231 posts

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#191

Post by hsivonen »

thaumasiotes wrote: post #181
jhanschoo wrote: post #170

Earlier quoted context omitted.

This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.

I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.

For writing globally appropriate software, UTF-8 is the best option that exists in practice.

Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#192

Post by jsjohnst »

knolax wrote: post #150
Avamander wrote: post #76

Earlier quoted context omitted.

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

I swear software's biggest problem is that developers prioritize their own ergonomics over actually producing functional code. Imagine if auto engineers just decided "safety is hard, let's just make deathtraps". I know the current narrative is that the 737 Max failed because of MBAs but it's a really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that cause…

> really big indictment that out of all the complex and hard to engineer systems in an aircraft it was poor software that caused the crash

While I frequently point out how bad we are as an industry at making fault tolerant code, this part is just flat wrong.

The software portion of the 737, while definitely flawed in a catastrophic way, would not have came to be if the aeronautics engineers had done their job and designed a flight worthy plane without software hacks. Not to imply the aeronautics guys are the root cause either though, the 737 Max fiasco is a top to bottom completely failure of Boeing as a whole, virtually every department involved in the Max has a significant reason to share in the blame.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#193

Post by camgunz »

lopsidedBrain wrote: post #121
camgunz wrote: post #89

Earlier quoted context omitted.

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. web…

I think we actually agree.

> A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless.

Totally agree, indexing into a string is bad practice, no matter the encoding (because you probably can't ever guarantee what the encoding will be, or how it was converted, etc.) This is true for UTF-8 and UTF-16, and really any encoding because again, you can't be sure what you're dealing with. Code points, "characters", glyphs, etc. are all concepts that work at different parts of the stack (well, characters doesn't), which again is true for all encodings.

The advantage UTF-16 has for languages that tend to be multibyte is it's representation takes up less space. Other than that, it has all the same disadvantages any other encoding has.

> Any kind of input-sanitization is vastly simplified in utf8, and that makes it worth it for me. For me the really troubling trends are conventions like Rust Utf8Error, where they can cause what I'd consider a UI-related exception in code that had no business even interpreting what those bytes are. Unfortunately, every API uses strings, so they are kind of hard to avoid. It introduces what I'd consider a software layering problem.

This is the problem right here. The "UTF-8" the world initiative ignores that UTF-16 is a lot more practical for most people, and as a result you get platforms expecting UTF-8 that really have no business doing so.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#194

Post by camgunz »

Freak_NL wrote: post #128
lopsidedBrain wrote: post #121

Earlier quoted context omitted.

I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. web…

Also, UTF-16 is not fixed width in modern usage. As soon as someone uses an emoji, boom! , surrogate pairs. So unless UTF-32 is used at a glorious four bytes per character, you won't get fixed width. > For me the really troubling trends are conventions like Rust Utf8Error,[…] Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? How is Rust's approach different from other langu…

Using any UTF encoding as fixed-width is almost always a mistake. There's no concept of "characters" in Unicode or the UTF encodings, they use codepoints, and those are only very rarely useful--you cannot treat them as "characters".

This is a common misconception with UTF encodings.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#195

Post by camgunz »

BoppreH wrote: post #92
camgunz wrote: post #89

Earlier quoted context omitted.

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point. It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...

Space is always a concern, and encodings should largely be transparent to your application unless you're writing something like a rendering engine.

Consider things like databases (relational/document/key-value/etc.), JSON payloads, compression algorithms, battery life, etc. etc.

There is a problem with lots of developers assuming UTF-16 is fixed-width (just look at this thread), but that's a mistake for any UTF encoding, and it's almost always a mistake in any encoding--you shouldn't be indexing into strings.

People say, "just use UTF-8, you never have to worry about it and it's ASCII compatible", but you shouldn't be using the ASCII compatibility (arguably this is an anti-feature) and you shouldn't have to worry about the details like BOM because you're using your language's built-in string support or a library right?

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#196

Post by camgunz »

jhanschoo wrote: post #101
camgunz wrote: post #89

Earlier quoted context omitted.

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.

There's a lot of text out there that isn't markup, or where markup is a sparse minority. Consider anything binary, ebooks, PDFs, various document formats, almost anything in a database, almost all messages sent via messaging apps, almost all emails, any JSON, anything in a protobuf, etc. etc. The vast majority of "text" in the world isn't in HTML/XML.

> When compression is involved, there is rarely a difference.

Rare is the situation where smaller input to a compression algorithm leads to larger output, and UTF-16 is smaller in lots of languages. It might rarely make a difference you, but that's not the same as rarely making a difference.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#197

Post by LeifCarrotson »

thaumasiotes wrote: post #157
gingerlime wrote: post #153

Earlier quoted context omitted.

Yes, you're absolutely right. The real bug was sending to the "wrong" matching email. But this is what makes this bug so hard to find. You're looking at "equal" strings, so why should it make a difference if you pick A or A if A === A ? :)

Hard to find, but it's the kind of thing that will hopefully come up in code review. Consider this pseudocode with helpful pseudo-hungarian notation: username = request.post_params('username') evil_email = request.post_params('email_address') user = get_user_by_name(username) good_email = user.email_address if good_email != evil_email: # Hackers! else: reset_password(user.id, evil_email) # it's fine; it's the same as…

The problem is that it requires a particularly detailed way of thinking about strings, much like the popular IEEE 754 floating point number question.

If the code were instead written:

  username = request.post_params('username')
  email = request.post_params('email_address')
  user = get_user_by_name(username)
 
  if emails_are_equal(email, user.email_address) 
    reset_password(user.id, email)
Most people would not feel compelled to use user.email_address in the call to reset_password, because they're guaranteed to be equal by the previous line! The vulnerability was introduced when the author of emails_are_equal() wrote it with conversion to uppercase of the whole thing, and failed to realize that this conversion can collide with Unicode domains. They may also have done things like normalize/remove optional.dots+suffixes@gmail.com, "quoted" or (commented) local parts. There's a lot of magic - and a lot of risk - in an emails_are_equal method.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#198

Post by Iv »

Avamander wrote: post #76
Iv wrote: post #59

Earlier quoted context omitted.

Disclaimer: as a heavy user of unicode, using it for both French and Japanese, I love it and see how important it is in the world. I just ranted about why ASCII is important to programmers: https://news.ycombinator.com/item?id=21760540 and this is a perfect example. ASCII has almost a 1-to-1 mapping between screen representation and byte representation. Once you know your font will differentiate between 1 i L | l and…

I'm personally incredibly annoyed by just the idea of "Unicode is hard, let's do ASCII", most of the world is non-ASCII, it's just annoying and sad to still see systems that fail when people try to use their native languages. UTF-8 should be the default pretty much everywhere, there are quite easy ways to avoid homograph attacks, those attacks are a poor excuse to discriminate against non-anglosphere.

I want to be able to collaborate with Japanese, Chinese, Indians, Russian, Arabian, Malay programmers. Even if we somehow live in a dystopia where all the english-speaking world disappears in a magic puff, I think I will continue to shut down my native french and communicate with other fellow programmers in English ASCII.

I wish more programmers understood the value of having a lingua franca is.

UTF-8 should be used everywhere on the user-facing side, but under the hoods, you want unambiguous representation of strings and code. It is easy to avoid homograph attacks if you do have a non-unicode representation available to you.

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#199

Post by Pahr3yah »

thristian wrote: post #71
dathinab wrote: post #57

Earlier quoted context omitted.

Note that Unicode did add a "uppercase-ish" ß, as it does appear in German, but only in context of an all caps word e.g on a sign board, so captilazation of a whole word to SS and that new all caps ß are both correct (not sure if UNICODE changed the capitalization rules or just added that strange all caps ß)

For backwards compatibility reasons, the capitalization rules can't be changed for existing characters. So normalizing by naive case-folding now requires at least three steps: "ẞ".to_lower() → "ß" "ß".to_upper() → "SS" "SS".to_lower() → "ss" (there's a standard for how to compare strings case-insensitively that doesn't involve repeated case-folding, but it's much more complex)

Unicode has a dedicated case folding[0][1] (as opposed to upper/lower case mapping) algorithm to cover these cases.

[0] https://www.unicode.org/reports/tr21/tr21-3.html#Caseless%20... [1] http://userguide.icu-project.org/transforms/casemappings#TOC...

Re: Hacking GitHub's Auth with Unicode's Turkish Dotless 'I'

#200

Post by Iv »

ComputerGuru wrote: post #102
tim333 wrote: post #81

Earlier quoted context omitted.

I was thinking that. Unicode is great for presenting text to users but for file names, email addresses, code and the like ASCII has a lot going for it.

No, it doesn’t. What’s a typical corporate email address? First.Last@...? F.Last@...? More than half the world’s population does not use the Latin alphabet.

Yes, and good luck using a search engine to find 月山＠何でも.jp or الله@gmail.com

By the way, the first one is not a mail address as ＠ and @ are different characters. But actually it may be because ＠ is the @ in the Japanese character set. So it probably should be accepted right? How about if I use the Japanese ＠ in a korean address? Is that valid? Or a likely attack?

Also in the Arabic name, the first character of the string is the one closer to the @ (it reads right to left), be sure to take that into account in a search interface.

I think it is totally utopian to think that a programmer can know all the subtleties of all the writing systems in the world. I agree that this is a problem that needs to be solved, but users and UX designers should in no way underestimate the magnitudes of additional work it causes.

Post reply on HN