Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS does not normalize Unicode filenames

mjtsai.com

Post reply on HN
Print view View original

111–120 of 148 posts

Re: APFS does not normalize Unicode filenames

#111

Post by cjensen »

ben_bai wrote: post #56
kalleboo wrote: post #32

Linus Thorvalds will be happy to hear that http://www.cio.com/article/2868393/linus-torvalds-apples-hfs...

HFS+ can be configured at creation time to be case sensitive. I did so a while back. Worked perfectly except for one application which could not find it's files. So i had to create a container and Format it case in sensitive and intall the APP there...

Adobe apps are notoriously incompatible with case sensitivity. If your apps work, that's great. But if Apple switched to case sensitivity by default, it would break apps.

Re: APFS does not normalize Unicode filenames

#112

Post by quotemstr »

Prego wrote: post #29

I've been very excited about ReFS -- a real "modern" Filesystem that leaves legacy issues behind. We've been using it for large storage systems, and am hoping it will become a viable solution for everything soon. It solves most of these issues.

A "modern" filesystem that lacks support for hard links? No thanks.

Re: APFS does not normalize Unicode filenames

#113

Post by jfim »

jstimpfle wrote: post #49
jfim wrote: post #13

Earlier quoted context omitted.

It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9). If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.

The problem exists already on the visual level, since there are pairs of distinct Unicode characters that look very much alike. I think the only thing you can really do about that is restricting to ASCII minus control characters. Probably many programmers would be willing to accept that, but non-technical users not. The next best thing would be to enforce a canonical (code-point?) encoding. But given the complexities…

> The next best thing would be to enforce a canonical (code-point?) encoding.

That's exactly what HFS+ did, and what APFS doesn't.

Re: APFS does not normalize Unicode filenames

#114

Post by hyperpape »

hawski wrote: post #73
jfim wrote: post #13

Earlier quoted context omitted.

It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9). If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.

Are you also against case sensitive file systems? Otherwise you can end up with two files that have the "same" name - eg. anne.jpg and Anne.jpg). Does normalization cover such a case?

For ascii, I would gladly do this There may be edge cases outside of the English language that I'm not aware of.

I am somewhat willing to accept the value of case sensitivity in identifiers for programming languages (though it's often related to the verbosity of the language, as in Java: "Foo foo = new Foo()"), but not in file naming.

Of course, I realize that the ship has sailed, and we're stuck with case sensitivity in file systems.

Re: APFS does not normalize Unicode filenames

#115

Post by bartvk »

djrogers wrote: post #12

iOS 10.3 with APFS has been in public and developer beta for several months - it's up to beta 7 right now in fact. If this were as vast a problem as Micheal Tsai presents in this post, wouldn't we (the devs and beta testers) be running in to this a lot? Given how loudly the tech press proclaims any perceived mis-step by Apple, I'd have to believe we'd have been reading tons of 'Apple is Doooooooomed' articles about t…

As I understand it, it may only be a problem when using the straight libc calls. But most devs just use the file system stuff that's in Apple's Foundation framework.

Re: APFS does not normalize Unicode filenames

#116

Post by kbenson »

comex wrote: post #33
ridiculous_fish wrote: post #14

Earlier quoted context omitted.

This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's b…

> What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Show two files with apparently identical names. Is that so surprising? There are many many ways for two different Unicode strings to look visually identical (or near-identical) even if they aren't equivalent under normalization. For…

> For that matter, even if you were limited to plain keyboard ASCII, you could have two really long filenames that only differ in one character, which are effectively indistinguishable without massive hair-pulling.

My initial response was to not present confusing data to the user, but then I thought of exactly this. In many fonts, lowercase "L" (l) and uppercase "i" (I) look the same. I'm sure in some fonts they actually are identical, rather than just being nearly indistinguishable to the human eye.

To the discussion is not about whether we should present confusing identifiers to users, but at what point if it becomes more common does it cause enough of a problem to be a bad choice, and what techniques are there to mitigate it?

Re: APFS does not normalize Unicode filenames

#117

Post by emn13 »

userbinator wrote: post #3

I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…

I'm not so sure: the thing is, those strings are not just a bag of bytes. They have semantics. Is it legal to include bytes that are 0? How about slashes in file names? How about byte sequences that aren't utf8 at all?

These things aren't just byte streams, they have semantics.

So one risk of treating this as "just bytes" is that bugs will introduce byte sequences that aren't utf8 at all, which will cause other programs to fail, or worse, to "try" and thereby corrupt the data further.

Another risk is that since it's supposed to be utf-8, some programs may do canonicalization internally to avoid confusing situations. This may even happen accidentally (though it's not likely): after all, a unicode-processing system could be forgiven for transparently changing canonicalization. But if a program canonicalizes you can now get really weird behavior such as opening a file, then saving it, and ending up with two files that look identical - because the write wrote to a path that was canonicalized.

Additionally, even though you can never avoid confusing paths entirely without considering the glyphs rendered, you are losing a very simple check against a fairly large class of errors.

Re: APFS does not normalize Unicode filenames

#118

Post by comex »

hamstergene wrote: post #104
QuercusMax wrote: post #4

This seems especially bad because US-based developers who don't test with unicode filenames might not come across this issue, leaving all their non-English-speaking customers broken. (Not that this excuses such developers in any way.) It also means that, yet again, every app will need to be updated for a new version of iOS. Makes me wonder how many apps will be left behind if not updated? Thousands? Hundreds of thous…

I'm going through apps on my phone and can hardly think what any of them would use Unicode filenames for. Say, a messenger might use user's nickname to name a history file — that would cause one-time loss of history, but not break the app. What else? Something tells me practically no apps will be seriously affected.

Any app that stores files in iCloud Drive has a fully user-accessible chunk of filesystem; it has to give the files meaningful names and properly handle files being renamed, added, etc. from the iCloud Drive app or file picker.

So they have to deal with arbitrary filenames; on the other hand, for the same reason, they can't maintain a master list of files somewhere which would break, but have to check the actual directory contents instead. Still, things like history or links between files might be broken.

That is, unless NSFileCoordination APIs act differently wrt normalization; iCloud clients have to use those rather than accessing the filesystem directly.

Re: APFS does not normalize Unicode filenames

#119

Post by eridius »

therealmarv wrote: post #91
eridius wrote: post #24

Earlier quoted context omitted.

What do you mean by "different"? Umlauts on HFS+ are still Umlauts anywhere else. The only real oddity of HFS+ (beyond the fact that it does normalization at all) is that it's not using NFD, it's using a variant of NFD based on an old version of Unicode (it has to be this way because the normalization tables must be immutable or there's compatibility issues when reading drives written to from different versions of th…

I don't know the technical details but the UTF-8 from a Mac is different from UTF-8 on Linux. Copy "hällö" from Mac to Linux over samba -> works fine. scp or rsync that same file again from Mac to Linux (without special conversion options) -> you will have two "hällö" there. The first one is recognized by Samba, the second one is invisible.

No, UTF-8 is the same everywhere. What you're confused about is the fact that for a lot of strings there are multiple different unicode scalar value sequences that represent the same user-visible string. This is the difference between composed and decomposed characters. For example, é can be represented either as a single scalar value that represents e-with-acute, or as two scalar values, a regular e followed by a combining acute mark.

On HFS+, because it normalizes, if you try to pass either form to the filesystem, it will treat them the same. But on filesystems that don't normalize (which is most of them) they'll be treated as distinct files. As a result, depending on the input you provide and whether the tools in question do any normalization, you could end up with two files that look identical (e.g. your two "hällö" files) but are different unicode sequences under the hood.

And none of this really has anything to do with UTF-8. UTF-8 is just a byte encoding scheme that can represent all valid unicode sequences.

Re: APFS does not normalize Unicode filenames

#120

Post by yuhong »

frik wrote: post #21
yuhong wrote: post #17

Earlier quoted context omitted.

That is due to the HFS+ normalization that APFS eliminates. The filename is stored in UTF-16 and OS X converts it to UTF-8.

Is it stored "real UTF-16" or in USC2 wide-chars like in NTFS? The later would be a big problem.

I believe HFS+ was created before Mac OS supported surrogates.
Post reply on HN