Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

141–150 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#141

Post by cryptonector »

konstmonst wrote: post #93
cryptonector wrote: post #78

I've been saying this for years: http://cryptonector.com/2010/04/on-unicode-normalization-or-... (originally at blogs.sun.com, now blogs.oracle.com, though I can't find it there). The problem is that most input methods produce something close to NFC while HFS+ decomposes to something close to NFD. Which means that if you cut-n-paste non-ASCII Unicode names from a finder into any app that doesn't normalize, then you'l…

I also think encoding doesn't belong into a file system. Let the names be arrays of bytes and leave the encoding to the people that use it, be it utf-8, utf-16 or something entirely different.

What really doesn't belong is keeping track of what codeset any given name/string uses. It's ETOOHARD.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#143

Post by cryptonector »

nottorp wrote: post #37
Piskvorrr wrote: post #29

Earlier quoted context omitted.

Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é ), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has…

Well, composed characters may or may not have been a good idea, although they're a pain for text search if you ask me. But why oh why did they have to specify FOUR normalization forms? Ref: http://www.unicode.org/reports/tr15/#Norm_Forms

Pre-composed characters were needed to make transcoding to ISO-8859-* fast.

Also, for Hangul, though decomposition is the better form to use normally (since it's phonetic, not syllabic), conversion to pre-composed syllabic form is very useful sometimes.

Even without pre-composition there would have been multiple equivalent forms of writing any character that requires more than one combining codepoint.

The only way to have avoided normalization altogether would have been to have no combining codepoints, and only a complete set of pre-compositions. This would have been less flexible, and very obnoxious for Hangul and possibly others.

I believe the need for normalization was simply unavoidable. It's not the fault of Unicode but the fault of humans' script designs. And it's OK.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#144

Post by cryptonector »

nottorp wrote: post #37
Piskvorrr wrote: post #29

Earlier quoted context omitted.

Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é ), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has…

Well, composed characters may or may not have been a good idea, although they're a pain for text search if you ask me. But why oh why did they have to specify FOUR normalization forms? Ref: http://www.unicode.org/reports/tr15/#Norm_Forms

You can ignore the K forms. So that's two forms, really, NFC and NFD.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#145

Post by cryptonector »

jrimbault wrote: post #18
Piskvorrr wrote: post #12

Earlier quoted context omitted.

Blame the encodings that came before it: Latin-1 and Windows-1252, and all the other ones. Hindsight is always 20/20, y'know.

I'm just being sarcastic: should we also blame the computers of old for not being able to handle more than 255 characters ?

No, just developers. Of course, the use of bytes for characters goes back a long time, to times when computers had small memories and disk (and other) storage capacities. And to even before then, to the days of telexes and typewriters. It's completely understandable. But UTF-8 is genius, which is why we use it.

Incidentally, ASCII was actually a multi-byte codeset... since one could combine most lower-case characters with BS (backspace) and overstrike with apostrophe, backtick, tilde, comma (for cedilles), and double-quotes (for umlauts), or with the same char for bold, or with underscore for underline. nroff(1) still uses this for bold and underscore, no?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#146

Post by cryptonector »

rurban wrote: post #73
Piskvorrr wrote: post #29

Earlier quoted context omitted.

Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é ), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has…

The biggest actual problem is Greek vs Cyrillic. They appear the same, but use different encodings. Any filename (directory entry as identifier) must forbid mixed scripts. See TR31 http://www.unicode.org/reports/tr31/ Combining marks or RTL switching tricks are the other popular spoofs.

Confusables are a real problem.

Some Cyrillic and Greek letters even look like Latin letters. And in some fonts '1' (one) and 'l' (ell) look the same even in Latin.

Dealing with these is much harder than with normalization forms. And confusables are a source of serious security headaches.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#147

Post by cryptonector »

Ono-Sendai wrote: post #54
TazeTSchnitzel wrote: post #47

Earlier quoted context omitted.

Compatibility and efficiency are two big reasons why you would want to have composed forms like e.g. the single codepoint for é: legacy encodings also have a single codepoint for it, and it's more efficient to represent it with one codepoint than with two. So, we need composed characters, but surely combining characters aren't needed, then? Well, no, because it's unreasonable to include a codepoint for every single c…

Combining characters seems like a pretty stupid thing to have in a character set/encoding. Does anyone know how many combined characters are actually needed?

More than there are pre-composed codepoint assignments for.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#148

Post by cryptonector »

TazeTSchnitzel wrote: post #47
nottorp wrote: post #7

To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several? Sounds like job security to me.

Compatibility and efficiency are two big reasons why you would want to have composed forms like e.g. the single codepoint for é: legacy encodings also have a single codepoint for it, and it's more efficient to represent it with one codepoint than with two. So, we need composed characters, but surely combining characters aren't needed, then? Well, no, because it's unreasonable to include a codepoint for every single c…

And also: scripts evolve. People may well start combining various diacritical marks in ways not done before.

Also, it can be useful to decompose in some cases. For example, it's useful for removing diacritical marks, which can be useful for fuzzy searches. Hangul is really phonetic, not syllabic, so there it's the reverse: decomposed forms are most useful, except when you need to think in syllables.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#149

Post by cryptonector »

tehabe wrote: post #30

Personally I think the current Normalisation Form D is awful, storing an ü as two characters is really annoying and even bash can't really deal with it in the version Apple uses. I really hope APFS will fix this. But we'll see.

It's not bash, BTW, but the OS X input mode. A real irony there.
Post reply on HN