Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

41–50 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#41
This is going to be fun with the java.nio.file API. Java currently normalises everything to NFC even though the file system on macOS is NFD. This was introduced some time in Java 8 because reasons.

Of course this is all going to break now, of course it will take years until Oracle fixes this.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#42
post #39

Earlier quoted context omitted.

By that logic that was true for HFS+ as well.

HFS+ did perform normalization on a file system level and stored strings as utf-16 so it's not true for HFS+.

Pretty sure normalization was done in open() and not the fs. Normalization was performed even for fat32 sticks.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#43
post #9

The real problem isn't whether filename normalization is a good/bad thing, the problem is Apple used to do it one way and is now switching without warning to doing it the other way . It's the logical end product of the odyssey from Apple's original philosophy of a resource and data fork model for files to the UNIX stream-of-bytes model for files. The UNIX model traditionally kept metadata about files separate (anyone…

Technically, .DS_Store is just Finder data - where is the windows displaying the directory, it's dimensions, listing mode, etc. The former resource fork is in ._$filename. You won't see it, unless you copy the file to smb share or zip it.

The former resource fork not in ._$filename unless the underlying filesystem does not support it. On HFS+ volumes, which is what 99%+ of Mac users use, there will be no ._$filename file.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#45
post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

NTFS, by comparison, does the "bag of wchar_t" instead of the "bag of byte" for filenames.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#46
post #22
post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

Normalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications. The comparison with other filesystems does not hold since applications for other OSs have always been developed with no normalization at FS level, and hence it was done by the applications, or through the use of high-level OS APIs. Mac applications, on the other hand, expect it to be the respo…

Do you have any recent benchmarks showing a significant impact from normalization? I haven't seen that on anything in at least a decade and that was simply Red Hat shipping an ancient and completely unoptimized libicu.

> Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article.

Again, the article made a huge sweeping claim without supporting it. That's simply not true in either way – many apps on every OS don't handle this at all, some handle it consistently everywhere, and what a “Mac application” means varies widely from “clean, modern Cocoa” to “uses a lot of C, etc. libraries”, “cross platform C++”, “C# port”, “Electron shell”, etc. You can't make any statement which is true for every single one of those categories, much less for every code path which eventually results in a filesystem call. I've run into cases where something mostly worked until you hit their integrated ZIP, Git/SVN, etc. support and found a new way that a filename was constructed.

My point wasn't that everything is fine but simply that this is complicated and no decision results in avoiding problems. Not normalizing allows for confusing visually-identical files; normalizing results in errors or data loss which will be blamed on the OS.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#47
post #7

To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several? Sounds like job security to me.

Compatibility and efficiency are two big reasons why you would want to have composed forms like e.g. the single codepoint for é: legacy encodings also have a single codepoint for it, and it's more efficient to represent it with one codepoint than with two.

So, we need composed characters, but surely combining characters aren't needed, then? Well, no, because it's unreasonable to include a codepoint for every single combination, especially once you have multiple diacritics on a single character, for example. Also, compatibility factors in here too, because legacy encodings likewise have combining characters.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#48

Earlier quoted context omitted.

Technically, .DS_Store is just Finder data - where is the windows displaying the directory, it's dimensions, listing mode, etc. The former resource fork is in ._$filename. You won't see it, unless you copy the file to smb share or zip it.

The former resource fork not in ._$filename unless the underlying filesystem does not support it. On HFS+ volumes, which is what 99%+ of Mac users use, there will be no ._$filename file.

You are correct regarding the resource fork support in the fs and it's visibility, maybe I didn't express myself as exactly as I should.

However, 99%+ of Mac users do use FAT-formatted USB sticks, ZIP files, or other fs/mechanism/whatever that does not support resource forks where the compatibility littering kicks in, so they, or the people they share their files with, will see ._$filename files too.

On the other hand, Finder litters with .DS_Store files everywhere, even on HFS+.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#49
post #37

Earlier quoted context omitted.

Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é ), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has…

Well, composed characters may or may not have been a good idea, although they're a pain for text search if you ask me. But why oh why did they have to specify FOUR normalization forms? Ref: http://www.unicode.org/reports/tr15/#Norm_Forms

Because you need a normalisation form fit for your particular purpose.
Post reply on HN