Can we believe our eyes? Misleading people with Unicode. • Hacker News Board

Re: Can we believe our eyes? Misleading people with Unicode.

#81

Post by Sephr » Thu, Aug 11, 2011, 6:31 PM UTC

cbr wrote: post #34

pilif wrote: post #27

Earlier quoted context omitted.

> A utf8 decoder should reject portions of utf8 streams that don't use the shortest possible encoding so you would say that there should be no file names using the cyrillic o? So if a russian-speaking person wants to save a file, that file name should be rejected? Or translated into a mish-mash between cyrillic and roman characters? How will that work if that filename is reused on a system on which the default font d…

> A utf8 decoder should reject portions of utf8 > streams that don't use the shortest possible encoding so you would say that there should be no file names using the cyrillic o? I'm sorry, I was unclear. I should have said "don't use the shortest possible encoding for a code point ". Cyrillic 'o' is code point U+043E while roman 'o' is code point U+006F. The canonicalization attack relies on overly liberal utf8 decod…

The Cyrillic o isn't being interpreted as U+006F, it just looks like it. A UTF-8 decoder doesn't know how it's output is going to be used.

Re: Can we believe our eyes? Misleading people with Unicode.

#82

Post by sesqu » Thu, Aug 11, 2011, 6:34 PM UTC

nodata wrote: post #11

Seems easy enough to guard against. Highlight the characters which are unexpected for my locale.

I wonder if it would suffice to highlight characters not found in the primary font. I don't know how, exactly, fonts fallthrough, but I doubt most fonts venture outside locales.

Re: Can we believe our eyes? Misleading people with Unicode.

#83

Post by driverdan » Thu, Aug 11, 2011, 6:37 PM UTC

driverdan wrote: post #68

I personally find the RLO / LRO issue much more concerning. I just tested Chrome and Firefox and found it works in URLs. You could rewrite pyapla.com to paypal.com and phish people easily.

I've done a bit more testing and found that while it can be used in URLs it's problematic. When pasted in the navigation bar it causes errors in both Chrome and Firefox. There's probably a way to exploit this and make it work but I don't have time to dig into it right now.

Re: Can we believe our eyes? Misleading people with Unicode.

#84

Post by sesqu » Thu, Aug 11, 2011, 6:43 PM UTC

Sephr wrote: post #81

cbr wrote: post #34

Earlier quoted context omitted.

> A utf8 decoder should reject portions of utf8 > streams that don't use the shortest possible encoding so you would say that there should be no file names using the cyrillic o? I'm sorry, I was unclear. I should have said "don't use the shortest possible encoding for a code point ". Cyrillic 'o' is code point U+043E while roman 'o' is code point U+006F. The canonicalization attack relies on overly liberal utf8 decod…

The Cyrillic o isn't being interpreted as U+006F, it just looks like it. A UTF-8 decoder doesn't know how it's output is going to be used.

That's not the point. The attack relies on some nonconforming decoders exhibiting a many-to-one mapping of bitstreams to codepoints, changing the semantics of the bistream.

Re: Can we believe our eyes? Misleading people with Unicode.

#85

Post by zbanks » Thu, Aug 11, 2011, 6:44 PM UTC

jerf wrote: post #42

yahelc wrote: post #5

Earlier quoted context omitted.

Whoa. Check out you did to the markup of this page. http://d.pr/HKSQ

The HTML standard specifies that changes in text direction are bounded to the block they occur in: http://www.w3.org/TR/1999/REC-html401-19991224/struct/dirlan... I discovered this as I was writing a paranoid HTML cleanup library and wanted to prevent the attack where a user sticks a text-direction-change character into the page and reverses the whole thing. As we've all just witnessed, that can't happen in a conform…

Doesn't the library just have to perform a simple search-and-replace on the entire page's HTML for the blacklisted characters to remove them?

Normally blacklisting is bad, but we're only targeting the few text-direction-changing characters that exist.

Re: Can we believe our eyes? Misleading people with Unicode.

#86

Post by cbr » Thu, Aug 11, 2011, 6:57 PM UTC

mcantor wrote: post #51

It's funny to think that a hapless vimmer who happens to be running Windows would have never noticed this, because they would simply have typed ":edit $SYSTEMROOT\system32\drivers\etc\hosts" and gotten the real file. (This isn't a "look how cool command line junkies are" comment; I was just musing.)

tab completion

Re: Can we believe our eyes? Misleading people with Unicode.

#87

Post by arvinjoar » Thu, Aug 11, 2011, 7:46 PM UTC

Seems like there's a hidden benefit to always keeping a tab with the hosts file open in notepad++. I use the hosts file from time to time, and I just leave the tab open, never thought it could help me out security-wise though.

Re: Can we believe our eyes? Misleading people with Unicode.

#88

Post by derobert » Thu, Aug 11, 2011, 7:51 PM UTC

sesqu wrote: post #82

nodata wrote: post #11

Seems easy enough to guard against. Highlight the characters which are unexpected for my locale.

I wonder if it would suffice to highlight characters not found in the primary font. I don't know how, exactly, fonts fallthrough, but I doubt most fonts venture outside locales.

Fonts increasingly venture outside of locales. Its very common for one font to have all western-European characters, or even all European (including Cyrillic) characters. This is especially true for the default system fonts.

CJK fonts often include not only CJK, but European characters as well. Almost always at least ASCII.

Ultimately, having a single font for all characters is desirable: Having to go track down more fonts because you're seeing � in your text is a pretty bad experience. Substituting other fonts is at best a kluge, as it often looks terrible.

Projects like DejaVu who plan to eventually cover all living scripts (http://dejavu-fonts.org/wiki/Plans) are not only a good thing but they are also making substantial progress.

Also, even in ASCII, there are a bunch of confusing characters (all depend on which font, of course): I (eye), l (ell), 1 (one), | (vertical bar); O (oh), and 0 (zero); {} (braces) and () (parentheses); 5 (five), S (ess), and $ (dollar), rn (r-n) and m; vv (v-v) and w; etc.

Wе nееԁ tօ fⅰnⅾ Ьеttеr ѕоⅼυtions tҺаɳ vіѕυаⅼⅼУ dіѕtіɳgυіѕҺing сҺаrаⅽtеrѕ.

Re: Can we believe our eyes? Misleading people with Unicode.

#89

Post by gwern » Thu, Aug 11, 2011, 7:54 PM UTC

I remember years back on Wikipedia, clever vandals would play Unicode tricks. It was interesting, to say the least - you'd register a name that looks identical to a real user, vandalize, and hope the administrator would type the name in...

This was ultimately stopped by Antispoof (https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extens...) but the bug reports are still interesting:

- https://bugzilla.wikimedia.org/show_bug.cgi?id=2593

- https://bugzilla.wikimedia.org/show_bug.cgi?id=2290

Re: Can we believe our eyes? Misleading people with Unicode.

#90

Post by andresmh » Thu, Aug 11, 2011, 7:55 PM UTC

njharman wrote: post #79

pavel_lishin wrote: post #69

Earlier quoted context omitted.

I wonder how/why people end up typing á in that URL - on my machine, it takes extra effort (OPTION-e a) to enter that.

non-english keyboards links

Yes. I think this is part of it. Here are the logs from a few weeks ago: http://j.mp/pBqfbx I'd be curious if anyone can figure out why so many visitors are using a browser called Netfront (apparently from Samsung mobiles).