Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

71–80 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#71

Post by mattnewton »

vetinari wrote: post #63
klodolph wrote: post #53

Earlier quoted context omitted.

Most files don't have resource forks though, so the ._$filename doesn't store resource fork data in most cases. It usually just stores metadata. I really doubt that 99%+ of Mac users use flash drives / zip files, though. I suspect that well more than 1% of users never use anything like that. A lot of people only have a single computer and share through e.g. just mailing files to people or using Dropbox, or not even t…

While resource forks are deprecated and on the retreat, extended attributes are a new hotness, used extensively (just download a file in the browser, and it will get com.apple.quarantine and com.apple.metadata:kMDItemWhereFroms). These are also shoveled into AppleDouble files. Actually, resource fork is just com.apple.ResourceFork extended attribute. I'm not sure that there are more than 1% of mac users do not exchan…

GP is saying they don't pass around flash drives to do so, not that they don't share files.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#73

Post by rurban »

Piskvorrr wrote: post #29
nottorp wrote: post #20

Earlier quoted context omitted.

I don't remember those having multiple representations for the same character in the same encoding or normal forms...

Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é ), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has…

The biggest actual problem is Greek vs Cyrillic. They appear the same, but use different encodings. Any filename (directory entry as identifier) must forbid mixed scripts. See TR31 http://www.unicode.org/reports/tr31/ Combining marks or RTL switching tricks are the other popular spoofs.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#74

Post by rurban »

netheril96 wrote: post #50

Do "most non-English" languages have normalization issues? At least CJK users do not.

Actually Korean (Hangul) is the biggest user of NFD normalization, with very special logic. Hangul also has the only 2 still remaining identifier bugs in the unicode 9 database. HANGUL FILLER and HALFWIDTH HANGUL FILLER wrongly being valid ID_Cont, as such wrongly usable as identifier characters (such as in filenames).

Almost every script but Latin-1 has TR31, TR36 and TR39 issues. http://www.unicode.org/reports/tr39/

Re: APFS is not safe to use with names which have Unicode normalisation issues

#75

Post by deathanatos »

asimpletune wrote: post #59

"Unusable" is a strong word to use. Should filesystems be making up for our Unicode shortcomings? From a SW design perspective, is that the most sensible place to pass the burden of responsibility? I would say that another way to handle it is to store a file name as an array of bytes and put the burden on software developers to interpret Unicode correctly. Swift does this pretty nicely. I would say the only downside…

> Should filesystems be making up for our Unicode shortcomings?

Absolutely, yes. File names are text by their very definition; that we've been treating them as "bags of bytes" is a historical tragedy. At the very least, file names need to be displayed, as text, to the user, so they should be stored as text, that is in some well-defined encoding, and yes, it should be the job of the filesystem driver / kernel to enforce that it's not writing garbage out to disk.

Reinventing that wheel in every system that in any way interacts with the filesystem is bad engineering, and doomed to fail.

Further, I don't see why the typical user should need to know or understand the differences between 'e\N{COMBINING ACUTE ACCENT}' and '\N{LATIN LOWERCASE E WITH ACUTE ACCENT}'. Likewise, I don't see why each and every piece of code should be forced to handle that. Developers will get this wrong. In fact, the article seems to say even Apple can't get it right, in that Finder will not correctly show the directory contents in some instances, and fails to open files in some instances, telling users the file "doesn't exist".

Re: APFS is not safe to use with names which have Unicode normalisation issues

#76

Post by Someone »

ioquatix wrote: post #4

The title is click-bait and over-dramatises the issue. The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). If you want to see the kind of issues that path normalisation brings, check out this: https://github.com/thibaudgg/rb-fsevent/blob/master/ext/fsev... I'd like to believe that most developer would prefer the current behaviour over t…

"The choice of APFS is that a filename is a sequence of bytes"

I don't think that is true. https://developer.apple.com/library/content/documentation/Fi...:

"Filenames in APFS are encoded in UTF-8 and aren’t normalized."

So, names must be UTF-8; not every sequence of bytes is valid UTF-8. What they removed from the file system are the Unicode normalization tables.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#77

Post by Someone »

bigin wrote: post #33
the_mitsuhiko wrote: post #27

Earlier quoted context omitted.

> The title is click-bait and over-dramatises the issue. Sadly it's not. > The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). That is incorrec. APFS treats filenames as utf-8 strings and depending on the API you are using normalization is still taking place but on different levels. For instance all Cocoa APIs will perform normalization…

So it is correct then, the file system doesn't concern itself with normalization. File names are stored internally as utf-8 strings which are just a sequence of bytes.

Every UTF-8 string is a sequence of bytes, but not every sequence of bytes is a UTF-8 string.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#78

Post by cryptonector »

I've been saying this for years: http://cryptonector.com/2010/04/on-unicode-normalization-or-... (originally at blogs.sun.com, now blogs.oracle.com, though I can't find it there).

The problem is that most input methods produce something close to NFC while HFS+ decomposes to something close to NFD. Which means that if you cut-n-paste non-ASCII Unicode names from a finder into any app that doesn't normalize, then you'll have problems.

The solution we came up with for ZFS was normalization-preserving but normalization-insensitive name comparison and directory hashing. This produces the best interoperability via NFS, SMB, WebDAV, and so on, and local POSIX access.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#79

Post by aidenn0 »

rbanffy wrote: post #65
klodolph wrote: post #45

Earlier quoted context omitted.

NTFS, by comparison, does the "bag of wchar_t" instead of the "bag of byte" for filenames.

> NTFS, by comparison, does the "bag of wchar_t" I threw up a little.

This is largely due to microsoft being an early adopter of unicode. In the 90s, unicode was limited to 16 bits, so switching to fixed-length 16-bit characters made a lot of sense.

Of course, it didn't take long for that to change, but MS had already put a lot of resources into switching to UCS-2.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#80

Post by acdha »

vintagedave wrote: post #51
acdha wrote: post #46

Earlier quoted context omitted.

Do you have any recent benchmarks showing a significant impact from normalization? I haven't seen that on anything in at least a decade and that was simply Red Hat shipping an ancient and completely unoptimized libicu. > Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article. Again, the article made a huge sweeping claim without suppo…

> Again, the article made a huge sweeping claim without supporting it. Looks to me like the article made claims and backed them up with examples and screenshots. Rather than saying the article is wrong, can you demonstrate /why/ it is wrong? Using its examples and concerns (Finder, console, and scripts)?

> Rather than saying the article is wrong, can you demonstrate /why/ it is wrong?

I think you might to re-read my entire comment: note that I'm not arguing that the technical details are wrong, only that they're insufficient to support the huge “APFS is unusable” conclusion.

As previously noted, Windows and Linux work the same way and they are used by more people in individual non-English locales than the total number of Mac users. Would you say “NTFS is unusable by non-English users” is a useful statement?

There's plenty of room to say that a particular tool needs improvement, or that people making systems which copy or archive files should check for pathological cases, but it doesn't help anything to overstate the case so broadly.

Post reply on HN