Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

51–60 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#51

Post by vintagedave »

acdha wrote: post #46
catwell wrote: post #22

Earlier quoted context omitted.

Normalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications. The comparison with other filesystems does not hold since applications for other OSs have always been developed with no normalization at FS level, and hence it was done by the applications, or through the use of high-level OS APIs. Mac applications, on the other hand, expect it to be the respo…

Do you have any recent benchmarks showing a significant impact from normalization? I haven't seen that on anything in at least a decade and that was simply Red Hat shipping an ancient and completely unoptimized libicu. > Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article. Again, the article made a huge sweeping claim without suppo…

> Again, the article made a huge sweeping claim without supporting it.

Looks to me like the article made claims and backed them up with examples and screenshots.

Rather than saying the article is wrong, can you demonstrate /why/ it is wrong? Using its examples and concerns (Finder, console, and scripts)?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#52

Post by the_mitsuhiko »

nottorp wrote: post #20
Piskvorrr wrote: post #12

Earlier quoted context omitted.

Blame the encodings that came before it: Latin-1 and Windows-1252, and all the other ones. Hindsight is always 20/20, y'know.

I don't remember those having multiple representations for the same character in the same encoding or normal forms...

> I don't remember those having multiple representations for the same character in the same encoding or normal forms...

But that's not really what is happing here. What is happening is that the renderer decides to merge some code points into one glyph. That was true long before unicode as well (for instance ligatures) or even on old terminals with overstriking. Unicode just made this more explicit.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#53

Post by klodolph »

vetinari wrote: post #48
klodolph wrote: post #43

Earlier quoted context omitted.

The former resource fork not in ._$filename unless the underlying filesystem does not support it. On HFS+ volumes, which is what 99%+ of Mac users use, there will be no ._$filename file.

You are correct regarding the resource fork support in the fs and it's visibility, maybe I didn't express myself as exactly as I should. However, 99%+ of Mac users do use FAT-formatted USB sticks, ZIP files, or other fs/mechanism/whatever that does not support resource forks where the compatibility littering kicks in, so they, or the people they share their files with, will see ._$filename files too. On the other han…

Most files don't have resource forks though, so the ._$filename doesn't store resource fork data in most cases. It usually just stores metadata.

I really doubt that 99%+ of Mac users use flash drives / zip files, though. I suspect that well more than 1% of users never use anything like that. A lot of people only have a single computer and share through e.g. just mailing files to people or using Dropbox, or not even that.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#54

Post by Ono-Sendai »

TazeTSchnitzel wrote: post #47
nottorp wrote: post #7

To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several? Sounds like job security to me.

Compatibility and efficiency are two big reasons why you would want to have composed forms like e.g. the single codepoint for é: legacy encodings also have a single codepoint for it, and it's more efficient to represent it with one codepoint than with two. So, we need composed characters, but surely combining characters aren't needed, then? Well, no, because it's unreasonable to include a codepoint for every single c…

Combining characters seems like a pretty stupid thing to have in a character set/encoding. Does anyone know how many combined characters are actually needed?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#55

Post by the_mitsuhiko »

nottorp wrote: post #37
Piskvorrr wrote: post #29

Earlier quoted context omitted.

Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é ), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has…

Well, composed characters may or may not have been a good idea, although they're a pain for text search if you ask me. But why oh why did they have to specify FOUR normalization forms? Ref: http://www.unicode.org/reports/tr15/#Norm_Forms

> But why oh why did they have to specify FOUR normalization forms?

Ignoring compatibility normalizations there are really only two and neither of those two you can remove. So that's a bit of an odd question. If you want to remove the compatibility forms that's fair but nobody is force to use them. You can consider them "partially normalized" or "not normalized at all" for all intents and purposes

Re: APFS is not safe to use with names which have Unicode normalisation issues

#56

Post by rndgermandude »

kalleboo wrote: post #5
ioquatix wrote: post #4

The title is click-bait and over-dramatises the issue. The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). If you want to see the kind of issues that path normalisation brings, check out this: https://github.com/thibaudgg/rb-fsevent/blob/master/ext/fsev... I'd like to believe that most developer would prefer the current behaviour over t…

The problem seems to come from the high-level APIs doing normalization. So if you have two files in a directory, one normalized and one not-normalized and open the non-normalized file, the high-level API will then normalize that filename and you open the wrong file. I've always disliked the practice of messing around with file paths (storing them, concatenating them, etc). I preferred the way that the Classic MacOS t…

Having two sets of APIs, one that does not mess with paths and one that does or adds other constraints, will cause all kinds of funky problems. For APFS it will be high-level-API using applications not being able to open certain files or using the wrong file[0], like stated.

You got a similar siltation with NTFS where the file system supports long paths while the userland WinAPI does use a much smaller length limit (unless you jump through "\\?\ hoops, which does not work for relative paths), which renders certain files "unopenable" by certain applications.

[0] Regarding opening the wrong files, anybody remember the Android zip vulnerabilities? https://googlesystem.blogspot.com/2013/07/the-8219321-androi... It's not that hard to imagine that some macOS software does security/sanity checks on files using the low-level non-normalized API but then opens the (wrong and unchecked) file later with normalizing high level API, or vice versa. Having this API discrepancy built into your OS certainly makes these kinds of things more likely.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#57

Post by alkonaut »

Wait why is a file name a sequence of bytes? Why is it not a sequence of bytes that form a valid string in some encoding, so that you can just refuse some file names? Who refuses invalid characters such as invisible characters or "/"? Would not the same place be a good place to refuse anything that doesn't normalize to the same byte sequence, or an even stricter subset?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#58

Post by loeg »

> Currently, the Mac Extended file system, HFS+, uses Normalisation Form D (NFD). Under that, é and é are automagically converted to é, and represented as three bytes, 65 cc 81.

This is actually an oversimplification. Mac NFD does not decompose characters in a few specific ranges[0]:

> U+2000 through U+2FFF, U+F900 through U+FAFF, and U+2F800 through U+2FAFF are not decomposed

But I do expect applications built for HFS+' pseudo-NFD to struggle with non-normalizing APFS.

[0]: https://developer.apple.com/library/content/qa/qa1173/_index...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#59

Post by asimpletune »

"Unusable" is a strong word to use. Should filesystems be making up for our Unicode shortcomings? From a SW design perspective, is that the most sensible place to pass the burden of responsibility? I would say that another way to handle it is to store a file name as an array of bytes and put the burden on software developers to interpret Unicode correctly. Swift does this pretty nicely.

I would say the only downside to this approach, is a user wouldn't be able to distinguish two files with the same name apart, but it's hard to imagine how they'd get to creating such a situation in the first place without the developer rule above being violated.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#60

Post by [deleted] »

the_mitsuhiko wrote: post #34
JoachimS wrote: post #31

Earlier quoted context omitted.

One of the new features in iOS 11 presented yesterday is the File App. It provides access to all files (at some level) in the system and on networked systems including iCloud, Dropbox etc.

That only ever surfaces filenames through cocoa APIs that underwent normalization.

[deleted]
Post reply on HN