Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS does not normalize Unicode filenames

mjtsai.com

Post reply on HN
Print view View original

11–20 of 148 posts

Re: APFS does not normalize Unicode filenames

#11

Post by saghm »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

> If you're really talking about bags (unordered sets), that would certainly make for an interesting filesystem since filename.txt, filemane.txt, and maletent.fix would all be the same...

I was thinking the same thing! Although if we're being pedantic, I think "bag" is an "unordered multiset". If it were just a plain unordered set, then "filename.txt" would also be the same as "filenam.txt".

Re: APFS does not normalize Unicode filenames

#12

Post by djrogers »

iOS 10.3 with APFS has been in public and developer beta for several months - it's up to beta 7 right now in fact. If this were as vast a problem as Micheal Tsai presents in this post, wouldn't we (the devs and beta testers) be running in to this a lot?

Given how loudly the tech press proclaims any perceived mis-step by Apple, I'd have to believe we'd have been reading tons of 'Apple is Doooooooomed' articles about this by now. Given that this hasn't happened, and I haven't seen similar problem reports on dev forums and other hangouts, I'd lean towards there being some miscommunication or misunderstanding here.

Re: APFS does not normalize Unicode filenames

#13

Post by jfim »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9).

If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.

Re: APFS does not normalize Unicode filenames

#14

Post by ridiculous_fish »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no...

The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's blessed paths /System/Library... and non-blessed paths like ~/Documents/Pokemon.txt. Addressing this properly means reifying that distinction: making apps always be explicit about whether they're working with the user's or the filesystem's view of a file.

While we're here, why the should the user even have to name every file? Do you name every piece of junk mail on your desk/kitchen island?

The ideal for the user is something like tagging, where naming stuff is optional, names are forgiving and perhaps not unique, and files are not restricted to being in one directory. Meanwhile, filesystem names are an implementation detail and the filesystem enjoys Unicode ignorance. Spotlight moved a bit in this direction, but the end goal is still awfully far away.

Re: APFS does not normalize Unicode filenames

#15

Post by silverwind »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

The same goes for case-sensitivity which, unfortunately, they see as a defect. It would be great for portability if APFS would stay case-insensitive like the Unix file systems, e.g. just a dumb layer that reads/writes bytes.

Re: APFS does not normalize Unicode filenames

#17

Post by yuhong »

therealmarv wrote: post #16

Is APFS still using Apple's style UTF-8 for e.g. Umlauts? I had a lot of trouble with rsync and also Samba later (filenames and folders hidden) when I discovered that Umlauts on HFS are different than Umlauts on e.g. Ext4.

That is due to the HFS+ normalization that APFS eliminates. The filename is stored in UTF-16 and OS X converts it to UTF-8.

Re: APFS does not normalize Unicode filenames

#19

Post by inimino »

silverwind wrote: post #15
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

The same goes for case-sensitivity which, unfortunately, they see as a defect. It would be great for portability if APFS would stay case-insensitive like the Unix file systems, e.g. just a dumb layer that reads/writes bytes.

I think you mean case-aware, not case-sensitive. Just bytes like Unix leads to case-sensitive behavior.

Re: APFS does not normalize Unicode filenames

#20

Post by Ericson2314 »

ridiculous_fish wrote: post #14
userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's b…

I'd argue: Use a langauge with generics and make the path type a ((de-)serializable) black box.

You still need to make a decision in the end, but are much more free to change it.

Post reply on HN