Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS is not safe to use with names which have Unicode normalisation issues

eclecticlight.co

Post reply on HN
Print view View original

131–140 of 191 posts

Re: APFS is not safe to use with names which have Unicode normalisation issues

#131

Post by klodolph »

mattnewton wrote: post #71
vetinari wrote: post #63

Earlier quoted context omitted.

While resource forks are deprecated and on the retreat, extended attributes are a new hotness, used extensively (just download a file in the browser, and it will get com.apple.quarantine and com.apple.metadata:kMDItemWhereFroms). These are also shoveled into AppleDouble files. Actually, resource fork is just com.apple.ResourceFork extended attribute. I'm not sure that there are more than 1% of mac users do not exchan…

GP is saying they don't pass around flash drives to do so, not that they don't share files.

I think that there are well more than 1% of users, maybe 20% or higher, that never share files except to attach a picture to an email, upload it to Facebook, or something similar like that. It's a bit of a nitpick, though, but we do often forget the "non-power" computer users.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#132

Post by throwaway2048 »

cryptonector wrote: post #101
ridiculous_fish wrote: post #88

Earlier quoted context omitted.

That's fascinating. What happens on lookup if there's two different files with canonically equivalent names?

Great question! You can't have two such files! Say you create a file with a name that has different NFC and NFD forms, and you create it with the NFC form. Then you go try to create it with the NFD form, well, if doing an exclusive create (O_EXCL) then you'll get EEXIST, else you'll open the existing file.

so it infact does do normalization on file create then? How is it managing this without doing so?

Re: APFS is not safe to use with names which have Unicode normalisation issues

#133

Post by callesgg »

cryptonector wrote: post #94
callesgg wrote: post #36

I dont expect my computer to treat charcters that look the same equally.

Even when they are the same character according to Unicode?! (I understand not treating confusable characters as the same. That's a different story.)

No, in my mind a filename should be an identifier made of a bunch of bytes. How you represent those bytes is not important in the context of the file subsystem.

For me it seems more like a problem with Unicode. I can see why it is the way it is from a certain perspective. Very connivent.

But it has broken the underlying abstraction layer.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#134

Post by [deleted] »

acdha wrote: post #81
the_mitsuhiko wrote: post #68

Earlier quoted context omitted.

> I've had to write code to deal with normalization changes in archives and it's quite tedious. You need to continue to write this on APFS. Nothing really changed in that regard. It's now just also possible to store denormalized filenames.

Exactly - and that matters to me in cases where the ability to say that someone copying files won't change the checksum on a manifest, which users find extremely confusing.

[deleted]

Re: APFS is not safe to use with names which have Unicode normalisation issues

#135

Post by cryptonector »

derefr wrote: post #125
cryptonector wrote: post #104

Earlier quoted context omitted.

Almost! But not quite: it aliases some possible filenames, though only they are valid UTF-8 encodings (or UTF-16 say, if one did this on Windows). In particular, it prevents the existence of two files with equivalent names.

In other words, like most RDBMSes, ZFS differentiates "field type" from "collation." "Field type" in an RDBMS controls what can be written (e.g. "valid UTF8 strings"), what will be read back (e.g. the use of the Unicode replacement character), and what special values like NULL will cast to. An RDBMS field's collation controls how values in the field will compare for equality, and what will happen when you sort on tha…

Well, readdir() in ZFS does not produce outputs in any order, since directories are just plain hash tables.

And the n-i string comparison function produces a boolean if I remember correctly, not a trinary. So it doesn't define a collation. But it could define a collation, that's true.

The problem with moving sorting into the kernel is that you now need to have the collations there (English? French? something else? "Unicode" is not enough), and user-land needs to tell the kernel what collation to use for any given process or thread or system call. That's ETOOMUCHWORK for everyone, so it doesn't happen.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#136

Post by cryptonector »

callesgg wrote: post #133
cryptonector wrote: post #94

Earlier quoted context omitted.

Even when they are the same character according to Unicode?! (I understand not treating confusable characters as the same. That's a different story.)

No, in my mind a filename should be an identifier made of a bunch of bytes. How you represent those bytes is not important in the context of the file subsystem. For me it seems more like a problem with Unicode. I can see why it is the way it is from a certain perspective. Very connivent. But it has broken the underlying abstraction layer.

There was no underlying layer, just lots of U.S.- or Western-centric assumptions. Unicode breaks them, but so did every codeset (Shift-JIS, this, that, and the other).

Re: APFS is not safe to use with names which have Unicode normalisation issues

#137

Post by cryptonector »

throwaway2048 wrote: post #132
cryptonector wrote: post #101

Earlier quoted context omitted.

Great question! You can't have two such files! Say you create a file with a name that has different NFC and NFD forms, and you create it with the NFC form. Then you go try to create it with the NFD form, well, if doing an exclusive create (O_EXCL) then you'll get EEXIST, else you'll open the existing file.

so it infact does do normalization on file create then? How is it managing this without doing so?

It normalizes on LOOKUP, not create. On create it preserves the original form. Think of this as like case-preserving/case-insensitive behavior, but for form rather than case.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#138

Post by cryptonector »

cryptonector wrote: post #78

I've been saying this for years: http://cryptonector.com/2010/04/on-unicode-normalization-or-... (originally at blogs.sun.com, now blogs.oracle.com, though I can't find it there). The problem is that most input methods produce something close to NFC while HFS+ decomposes to something close to NFD. Which means that if you cut-n-paste non-ASCII Unicode names from a finder into any app that doesn't normalize, then you'l…

See also: http://cryptonector.com/2006/12/filesystem-i18n/

Re: APFS is not safe to use with names which have Unicode normalisation issues

#139

Post by cryptonector »

acdha wrote: post #128
cryptonector wrote: post #82

Earlier quoted context omitted.

Not true! ZFS also normalizes. However, while HFS+ normalizes on _create_ (bad), ZFS normalizes on lookup (good). Specifically, ZFS has a normalization-preserving, normalization-insensitive behavior -- a lot like case-preserving but case-insensitive behavior, but for normalization forms rather than case. The way this works is that there's a) a string comparison function that can provide normalization- and/or case-ins…

> Not true! ZFS also normalizes. However, while HFS+ normalizes on _create_ (bad), ZFS normalizes on lookup (good). That's not a question of being true or not but referring to different things. The distinction I was trying to make is that HFS+ will force every filename into NFD. With ZFS, the filename as received from the APIs should be the same byte sequence which was used to create it and that avoids an entire cate…

Yes, exactly. You can't have those buggy behaviors in git/svn/hg/whatever if the FS is form-preserving.

Re: APFS is not safe to use with names which have Unicode normalisation issues

#140

Post by cryptonector »

burnbabyburn wrote: post #44

I really fail to see why do you want your file system to normalize utf8 chars with hfs+ rules.

ISTR an old blog post or doc about this. Apple engineers wanted to be able to do prefix matching in HFS+, if I remember correctly. And so making the name as long as possible (via canonical decomposition) made sense to them.
Post reply on HN