Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

101–110 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#101

Post by cryptonector »

ridiculous_fish wrote: post #88
cryptonector wrote: post #82

Earlier quoted context omitted.

Not true! ZFS also normalizes. However, while HFS+ normalizes on _create_ (bad), ZFS normalizes on lookup (good). Specifically, ZFS has a normalization-preserving, normalization-insensitive behavior -- a lot like case-preserving but case-insensitive behavior, but for normalization forms rather than case. The way this works is that there's a) a string comparison function that can provide normalization- and/or case-ins…

That's fascinating. What happens on lookup if there's two different files with canonically equivalent names?

Great question! You can't have two such files!

Say you create a file with a name that has different NFC and NFD forms, and you create it with the NFC form. Then you go try to create it with the NFD form, well, if doing an exclusive create (O_EXCL) then you'll get EEXIST, else you'll open the existing file.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#103

Post by foota »

cryptonector wrote: post #95
mchanson wrote: post #23

Earlier quoted context omitted.

"without warning"? Not really. This has been discussed to death on ATP last year. Also, in regards to the headline, there are tens or maybe over a hundred million non-english speakers using iOS already running APFS...

Did anyone mention normalization-preserving/normalization-insensitive behavior? That's what ZFS does, and it's great!

Is that not just treating the filename as a byte sequence?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#104

Post by cryptonector »

foota wrote: post #103
cryptonector wrote: post #95

Earlier quoted context omitted.

Did anyone mention normalization-preserving/normalization-insensitive behavior? That's what ZFS does, and it's great!

Is that not just treating the filename as a byte sequence?

Almost! But not quite: it aliases some possible filenames, though only they are valid UTF-8 encodings (or UTF-16 say, if one did this on Windows). In particular, it prevents the existence of two files with equivalent names.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#105

Post by foota »

cryptonector wrote: post #104
foota wrote: post #103

Earlier quoted context omitted.

Is that not just treating the filename as a byte sequence?

Almost! But not quite: it aliases some possible filenames, though only they are valid UTF-8 encodings (or UTF-16 say, if one did this on Windows). In particular, it prevents the existence of two files with equivalent names.

Interesting, so it recognizes different encodings of the same Unicode code points?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#106

Post by foota »

foota wrote: post #105
cryptonector wrote: post #104

Earlier quoted context omitted.

Almost! But not quite: it aliases some possible filenames, though only they are valid UTF-8 encodings (or UTF-16 say, if one did this on Windows). In particular, it prevents the existence of two files with equivalent names.

Interesting, so it recognizes different encodings of the same Unicode code points?

Or does it just map code points that are identical to each other?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#107

Post by cryptonector »

aidenn0 wrote: post #79
rbanffy wrote: post #65

Earlier quoted context omitted.

> NTFS, by comparison, does the "bag of wchar_t" I threw up a little.

This is largely due to microsoft being an early adopter of unicode. In the 90s, unicode was limited to 16 bits, so switching to fixed-length 16-bit characters made a lot of sense. Of course, it didn't take long for that to change, but MS had already put a lot of resources into switching to UCS-2.

HFS+'s use of NFD also comes from it being an early adopter of Unicode.

When I argued strenuously for n-p/n-i behavior in ZFS (see elsewhere in this thread) I was told that gee, Unicode doesn't describe n-i string comparison the way I was proposing, so we can't do it :( However, fast n-i does follow from the spec, so eventually it got done. I can't take credit for the code, though I did contribute an optimization.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#108

Post by cryptonector »

pilif wrote: post #21

HFS+ was the only file system I know of that was doing Unicode normalisation and is certainly was the only one choosing NFD which encodes characters in a way that's impossible for someone to easily type in the UI. Short-term this will cause a lot of inconsistencies with applications using low-level APIs as the files currently existing will be NFD normalised, but any user-given path will very likely be more or less eq…

Not to repeat myself too much, but... ZFS also supports normalization. Only instead of normalizing on create, it has normalization-preserving/insensitive behavior.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#109

Post by cryptonector »

asimpletune wrote: post #59

"Unusable" is a strong word to use. Should filesystems be making up for our Unicode shortcomings? From a SW design perspective, is that the most sensible place to pass the burden of responsibility? I would say that another way to handle it is to store a file name as an array of bytes and put the burden on software developers to interpret Unicode correctly. Swift does this pretty nicely. I would say the only downside…

> Should filesystems be making up for our Unicode shortcomings?

In this case: yes. Specifically the FS should implement normalization-preserving/insensitive behavior. http://cryptonector.com/2010/04/on-unicode-normalization-or-...

Re: APFS is not safe to use with names which have Unicode normalisation issues

#110

Post by cryptonector »

konstmonst wrote: post #97
deathanatos wrote: post #75

Earlier quoted context omitted.

> Should filesystems be making up for our Unicode shortcomings? Absolutely, yes. File names are text by their very definition; that we've been treating them as "bags of bytes" is a historical tragedy. At the very least, file names need to be displayed, as text, to the user, so they should be stored as text , that is in some well-defined encoding, and yes, it should be the job of the filesystem driver / kernel to enfo…

But what is text? Not everyone wants to use unicode. It is dependent of the platform, the region, the OS and on many other different things like LC_* variables on linux. Why should a filesystem depend on those too?

ZFS has a per-dataset option to allow/reject non-UTF-8. For valid UTF-8 names ZFS implements normalization-preserving/insensitive behavior. That was the best compromise we could find, and it works really well.
Post reply on HN