Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

91–100 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#91

Post by cryptonector »

rurban wrote: post #74
netheril96 wrote: post #50

Do "most non-English" languages have normalization issues? At least CJK users do not.

Actually Korean (Hangul) is the biggest user of NFD normalization, with very special logic. Hangul also has the only 2 still remaining identifier bugs in the unicode 9 database. HANGUL FILLER and HALFWIDTH HANGUL FILLER wrongly being valid ID_Cont, as such wrongly usable as identifier characters (such as in filenames). Almost every script but Latin-1 has TR31, TR36 and TR39 issues. http://www.unicode.org/reports/tr39…

Hangul has composed and decomposed forms, but only the decomposed forms are used as canonical, even in NFC.

(Incidentally, NFC is closed to new compositions, though new compositions can be added to Unicode.)

Re: APFS is not safe to use with names which have Unicode normalisation issues

#92

Post by cryptonector »

loeg wrote: post #62
burnbabyburn wrote: post #44

I really fail to see why do you want your file system to normalize utf8 chars with hfs+ rules.

Normalizing to NFC form would make some sense, and was conceivably an option.

Normalization-preserving/normalization-insensitive behavior makes even more sense!

Re: APFS is not safe to use with names which have Unicode normalisation issues

#93

Post by konstmonst »

cryptonector wrote: post #78

I've been saying this for years: http://cryptonector.com/2010/04/on-unicode-normalization-or-... (originally at blogs.sun.com, now blogs.oracle.com, though I can't find it there). The problem is that most input methods produce something close to NFC while HFS+ decomposes to something close to NFD. Which means that if you cut-n-paste non-ASCII Unicode names from a finder into any app that doesn't normalize, then you'l…

I also think encoding doesn't belong into a file system. Let the names be arrays of bytes and leave the encoding to the people that use it, be it utf-8, utf-16 or something entirely different.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#94

Post by cryptonector »

callesgg wrote: post #36

I dont expect my computer to treat charcters that look the same equally.

Even when they are the same character according to Unicode?!

(I understand not treating confusable characters as the same. That's a different story.)

Re: APFS is not safe to use with names which have Unicode normalisation issues

#95

Post by cryptonector »

mchanson wrote: post #23
cstross wrote: post #9

The real problem isn't whether filename normalization is a good/bad thing, the problem is Apple used to do it one way and is now switching without warning to doing it the other way . It's the logical end product of the odyssey from Apple's original philosophy of a resource and data fork model for files to the UNIX stream-of-bytes model for files. The UNIX model traditionally kept metadata about files separate (anyone…

"without warning"? Not really. This has been discussed to death on ATP last year. Also, in regards to the headline, there are tens or maybe over a hundred million non-english speakers using iOS already running APFS...

Did anyone mention normalization-preserving/normalization-insensitive behavior? That's what ZFS does, and it's great!

Re: APFS is not safe to use with names which have Unicode normalisation issues

#96

Post by cryptonector »

klodolph wrote: post #45
acdha wrote: post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

NTFS, by comparison, does the "bag of wchar_t" instead of the "bag of byte" for filenames.

"Bag of code units" would be a better description (if it's correct anyways).

This is not very good. You want normalization-preserving/insensitive behavior instead.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#97

Post by konstmonst »

deathanatos wrote: post #75
asimpletune wrote: post #59

"Unusable" is a strong word to use. Should filesystems be making up for our Unicode shortcomings? From a SW design perspective, is that the most sensible place to pass the burden of responsibility? I would say that another way to handle it is to store a file name as an array of bytes and put the burden on software developers to interpret Unicode correctly. Swift does this pretty nicely. I would say the only downside…

> Should filesystems be making up for our Unicode shortcomings? Absolutely, yes. File names are text by their very definition; that we've been treating them as "bags of bytes" is a historical tragedy. At the very least, file names need to be displayed, as text, to the user, so they should be stored as text , that is in some well-defined encoding, and yes, it should be the job of the filesystem driver / kernel to enfo…

But what is text? Not everyone wants to use unicode. It is dependent of the platform, the region, the OS and on many other different things like LC_* variables on linux. Why should a filesystem depend on those too?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#98

Post by dietrichepp »

aidenn0 wrote: post #79
rbanffy wrote: post #65

Earlier quoted context omitted.

> NTFS, by comparison, does the "bag of wchar_t" I threw up a little.

This is largely due to microsoft being an early adopter of unicode. In the 90s, unicode was limited to 16 bits, so switching to fixed-length 16-bit characters made a lot of sense. Of course, it didn't take long for that to change, but MS had already put a lot of resources into switching to UCS-2.

This timeline also meant that wchar_t was fixed as 16 bits on Windows for compatibility, but ended up as 32 bits on macOS and Unix. It can't really be removed from the C or C++ standards, but there's nothing you can do with wchar_t that's both useful and portable.

Well-meaning developers learn how to support Unicode with wchar_t only to discover that wchar_t is the worst way to support Unicode.

https://www.moria.us/articles/wchar-is-a-historical-accident...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#99

Post by cryptonector »

konstmonst wrote: post #93
cryptonector wrote: post #78

I've been saying this for years: http://cryptonector.com/2010/04/on-unicode-normalization-or-... (originally at blogs.sun.com, now blogs.oracle.com, though I can't find it there). The problem is that most input methods produce something close to NFC while HFS+ decomposes to something close to NFD. Which means that if you cut-n-paste non-ASCII Unicode names from a finder into any app that doesn't normalize, then you'l…

I also think encoding doesn't belong into a file system. Let the names be arrays of bytes and leave the encoding to the people that use it, be it utf-8, utf-16 or something entirely different.

ZFS has an option to reject non-UTF-8. But if you disable that then it allows it, and for all UTF-8 it is normalization-preserving/insensitive. This is cheap and ideal.

EDIT: i.e., ZFS doesn't know or care about encodings. Its n-p/n-i behavior means that for non-UTF-8 that happens to appear as valid UTF-8 there is some potential aliasing behavior going on, but this is exceedingly unlikely, and we decided that it was worth taking that risk.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#100

Post by cryptonector »

ioquatix wrote: post #4

The title is click-bait and over-dramatises the issue. The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). If you want to see the kind of issues that path normalisation brings, check out this: https://github.com/thibaudgg/rb-fsevent/blob/master/ext/fsev... I'd like to believe that most developer would prefer the current behaviour over t…

The problem is serious if you run into it.

The FS needs to be normalization-preserving/insensitive. http://cryptonector.com/2010/04/on-unicode-normalization-or-...

Post reply on HN