Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

21–30 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#21

Post by pilif »

HFS+ was the only file system I know of that was doing Unicode normalisation and is certainly was the only one choosing NFD which encodes characters in a way that's impossible for someone to easily type in the UI.

Short-term this will cause a lot of inconsistencies with applications using low-level APIs as the files currently existing will be NFD normalised, but any user-given path will very likely be more or less equivalent to what NFC would do.

So in the short term, this will be a mess (unless the APFS conversion on install-time also does a once-time conversion of the normal form), but in the long term, I believe this is the way to go.

And for that matter, the same should happen with regards to case-insensitivity which is even worse as ensuring case-insensitivity might actually be dependent on locale, which means that depending on the user's locale two file names might or might not be identical under case-insensitivity rules.

Unfortunately, it look like there's just too much legacy code around that plain doesn't work with a case-sensitive file-system on the mac (I'm looking at you, Adobe).

BTW: On my 10.13 Beta 1 setup with an APFS converted boot drive, unless I manually create a NFD encoded file name on the command line (which you can't do accidentally), everything is NFC both in the UI and on the command line. This also applies to files I haven't touched since the conversion to APFS.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#22

Post by catwell »

acdha wrote: post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

Normalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications.

The comparison with other filesystems does not hold since applications for other OSs have always been developed with no normalization at FS level, and hence it was done by the applications, or through the use of high-level OS APIs. Mac applications, on the other hand, expect it to be the responsibility of the filesystem. This is explained pretty clearly in the article.

On a side note, I wish Apple would have taken this opportunity to switch normalization from NFD to NFC, which basically everything else uses. The distinction causes complexity and often issues in software which share data between Apple platforms and other platforms, such as version control systems for instance.

EDIT: according to pilif's comment, they did, which is awesome!

Re: APFS is not safe to use with names which have Unicode normalisation issues

#23

Post by mchanson »

cstross wrote: post #9

The real problem isn't whether filename normalization is a good/bad thing, the problem is Apple used to do it one way and is now switching without warning to doing it the other way . It's the logical end product of the odyssey from Apple's original philosophy of a resource and data fork model for files to the UNIX stream-of-bytes model for files. The UNIX model traditionally kept metadata about files separate (anyone…

"without warning"? Not really. This has been discussed to death on ATP last year.

Also, in regards to the headline, there are tens or maybe over a hundred million non-english speakers using iOS already running APFS...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#24

Post by TorKlingberg »

mchanson wrote: post #23
cstross wrote: post #9

The real problem isn't whether filename normalization is a good/bad thing, the problem is Apple used to do it one way and is now switching without warning to doing it the other way . It's the logical end product of the odyssey from Apple's original philosophy of a resource and data fork model for files to the UNIX stream-of-bytes model for files. The UNIX model traditionally kept metadata about files separate (anyone…

"without warning"? Not really. This has been discussed to death on ATP last year. Also, in regards to the headline, there are tens or maybe over a hundred million non-english speakers using iOS already running APFS...

On iOS the user doesn't get much direct access to the file system.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#25

Post by the_mitsuhiko »

Sadly this was expected but it's not clear what Apple wants developers to do about this now. I am very happy that normalization is removed from the FS layer but I wish they had removed it from Cocoa as well or at least picked a different normalization form.

The fact that Cocoa apps create different filenames than the terminal is not great. It's even worse that some UI (like Finder) seem to cause even more confusion is not helping.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#26

Post by the_mitsuhiko »

mchanson wrote: post #23
cstross wrote: post #9

The real problem isn't whether filename normalization is a good/bad thing, the problem is Apple used to do it one way and is now switching without warning to doing it the other way . It's the logical end product of the odyssey from Apple's original philosophy of a resource and data fork model for files to the UNIX stream-of-bytes model for files. The UNIX model traditionally kept metadata about files separate (anyone…

"without warning"? Not really. This has been discussed to death on ATP last year. Also, in regards to the headline, there are tens or maybe over a hundred million non-english speakers using iOS already running APFS...

iOS never exposes low-level filenames to the user.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#27

Post by the_mitsuhiko »

ioquatix wrote: post #4

The title is click-bait and over-dramatises the issue. The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). If you want to see the kind of issues that path normalisation brings, check out this: https://github.com/thibaudgg/rb-fsevent/blob/master/ext/fsev... I'd like to believe that most developer would prefer the current behaviour over t…

> The title is click-bait and over-dramatises the issue.

Sadly it's not.

> The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here).

That is incorrec. APFS treats filenames as utf-8 strings and depending on the API you are using normalization is still taking place but on different levels. For instance all Cocoa APIs will perform normalization but the underlying syscalls and posix APIs will not.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#28

Post by the_mitsuhiko »

nottorp wrote: post #7

To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several? Sounds like job security to me.

> To be honest, I blame Unicode. Why allow different representations for the same character, and then provide a normalized form anyway, except it's not one normalized form but several?

That's not really the issue here. Even if you can avoid normalization in some languages the problem will come back in others. For instance what are you going to do about invisible characters, control characters or more? Traditionally software attempted to just ignore the problems and let it blow up in other places. For some fun issues just try whitespace and bell characters in filenames and navigate around in your shell.

Normalization is just one of many issues with filenames.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#29

Post by Piskvorrr »

nottorp wrote: post #20
Piskvorrr wrote: post #12

Earlier quoted context omitted.

Blame the encodings that came before it: Latin-1 and Windows-1252, and all the other ones. Hindsight is always 20/20, y'know.

I don't remember those having multiple representations for the same character in the same encoding or normal forms...

Indeed not: there is an almost infinite number of things that Latin-1 did not have; e.g. any characters beyond Western Europe. While keeping the 128-256 block compatible was a part of early Unicode (hence the "one-glyph" é), having composed characters was a Unicode primary design goal (hence e and the composing accent). A pure Unicode implementation would have been better, maybe; what we have instead is one that has gained mass adoption.

https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_bl...

Post reply on HN