Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

81–90 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#81

Post by acdha »

the_mitsuhiko wrote: post #68
acdha wrote: post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

> I've had to write code to deal with normalization changes in archives and it's quite tedious. You need to continue to write this on APFS. Nothing really changed in that regard. It's now just also possible to store denormalized filenames.

Exactly - and that matters to me in cases where the ability to say that someone copying files won't change the checksum on a manifest, which users find extremely confusing.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#82

Post by cryptonector »

acdha wrote: post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

Not true! ZFS also normalizes. However, while HFS+ normalizes on _create_ (bad), ZFS normalizes on lookup (good).

Specifically, ZFS has a normalization-preserving, normalization-insensitive behavior -- a lot like case-preserving but case-insensitive behavior, but for normalization forms rather than case.

The way this works is that there's a) a string comparison function that can provide normalization- and/or case-insensitive comparisons, b) a character-at-a-time normalization function of sorts used for directory hashing (since ZFS hashes directories).

This is much better than normalizing on create, which is destructive and obnoxious when the form you normalize to is not the more common form and you live in a sea of applications that don't do normalization.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#83

Post by cryptonector »

catwell wrote: post #22
acdha wrote: post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

Normalization is very expensive and does not belong at the FS level. This kills performance for some classes of applications. The comparison with other filesystems does not hold since applications for other OSs have always been developed with no normalization at FS level, and hence it was done by the applications, or through the use of high-level OS APIs. Mac applications, on the other hand, expect it to be the respo…

Normalization is not that expensive if you only care to do normalization-insensitive string comparison and string hashing.

The reason it's not that expensive is: a) this requires no memory allocation, and b) most characters in most strings require no normalization!

Notionally you just look at pairs of next codepoints, and if the second one isn't combining and the first one is canonical for the chosen NF (a very fast check for ASCII!), then there's no need to normalize the first, otherwise you gather the combining codepoints and normalize, producing one normalized character and restarting the process where you left off. Most of the time the first codepoint requires no normalization, so the fast path is fast -- not as fast as a normal strcmp() or memcmp(), but still pretty fast.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#84

Post by cryptonector »

xoa wrote: post #64
pilif wrote: post #21

HFS+ was the only file system I know of that was doing Unicode normalisation and is certainly was the only one choosing NFD which encodes characters in a way that's impossible for someone to easily type in the UI. Short-term this will cause a lot of inconsistencies with applications using low-level APIs as the files currently existing will be NFD normalised, but any user-given path will very likely be more or less eq…

> HFS+ was the only file system I know of that was doing Unicode normalisation I've seen something in a couple comments now to this effect, does nobody here do anything with ZFS at all? It's a pretty great filesystem on not just illumos but FreeBSD, Linux, and macOS, and it has full native normalization support (and for all 4 forms too). Normalizing is optional, but it's definitely there and with Macs I have been usi…

Thank you for pointing this out.

In particular ZFS has normalization-preserving/normalization-insensitive behavior, which is far superior to HFS+'s opinionated normalization-on-create (to a form that is different from the common input modes' output!).

Re: APFS is not safe to use with names which have Unicode normalisation issues

#85

Post by cryptonector »

bigin wrote: post #33
the_mitsuhiko wrote: post #27

Earlier quoted context omitted.

> The title is click-bait and over-dramatises the issue. Sadly it's not. > The choice of APFS is that a filename is a sequence of bytes. Nothing more, nothing less (feel free to correct me if I'm wrong here). That is incorrec. APFS treats filenames as utf-8 strings and depending on the API you are using normalization is still taking place but on different levels. For instance all Cocoa APIs will perform normalization…

So it is correct then, the file system doesn't concern itself with normalization. File names are stored internally as utf-8 strings which are just a sequence of bytes.

A terrible mistake.

APFS should have been normalization-preserving/normalization-insensitive.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#86

Post by cryptonector »

_jomo wrote: post #19

The actual problem is not APFS but Apple's programs doing (and their advice to developers to do) normalization. It wouldn't be a problem if programs just used the file names given to them. In fact, I think the normalization HFS+ does is more problematic. For example, fish shell can't complete file names when you use un-normalized characters in the input. [0][1] Edit: 0: "Unicode normalization issues with HFS+" https:…

http://cryptonector.com/2010/04/on-unicode-normalization-or-...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#87

Post by cryptonector »

kuon wrote: post #6

I'm happy that the filesystem treats name as sequences of bytes. Normalization should happen at a higher level.

I'm not. The FS should be normalization-preserving/normalization-insensitive! http://cryptonector.com/2010/04/on-unicode-normalization-or-...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#88

Post by ridiculous_fish »

cryptonector wrote: post #82
acdha wrote: post #10

There's a potentially useful discussion to be had on normalization but the title is pure clickbait hyperbole. HFS+ is the only filesystem in common use which performs Unicode normalization and a statement that bold would require at least some evidence that Windows, Linux, etc. are only usable by English speakers. My position on this is mixed. I've had to write code to deal with normalization changes in archives and i…

Not true! ZFS also normalizes. However, while HFS+ normalizes on _create_ (bad), ZFS normalizes on lookup (good). Specifically, ZFS has a normalization-preserving, normalization-insensitive behavior -- a lot like case-preserving but case-insensitive behavior, but for normalization forms rather than case. The way this works is that there's a) a string comparison function that can provide normalization- and/or case-ins…

That's fascinating. What happens on lookup if there's two different files with canonically equivalent names?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#89

Post by cryptonector »

markonen wrote: post #17

Apparently on iOS 11, even the case-sensitive variant of APFS will be normalization-insensitive. Previously it looked like this would only be the case on macOS's case-insensitive APFS variant. Anyway, it looks like the issues raised in this (April) blog post will not actually apply to iOS 11 or macOS High Sierra.

Oh, that's great!

I've been saying for years, to anyone who will listen (and many who won't!) that normalization-preserving/normalization-insensitive is the only sensible behavior for a filesystem.

(I've said this many times in the IETF in the context of NFSv4 and WebDAV and such. Every time I've noticed the subject of stringprep for filesystem protocols come up.)

http://cryptonector.com/2010/04/on-unicode-normalization-or-...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#90

Post by cryptonector »

tehabe wrote: post #30

Personally I think the current Normalisation Form D is awful, storing an ü as two characters is really annoying and even bash can't really deal with it in the version Apple uses. I really hope APFS will fix this. But we'll see.

It's awful in great part because common input modes produce something close to NFC.
Post reply on HN