Earlier quoted context omitted.
You don't, there is only one unicode representation of those characters.
So what does mac do with those ?
Use that representation? The problem only happens when there is more than one way to represent the same text.
APFS does not normalize Unicode filenames
61–70 of 148 posts
Re: APFS does not normalize Unicode filenames
#62Earlier quoted context omitted.
This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's b…
I'd argue: Use a langauge with generics and make the path type a ((de-)serializable) black box. You still need to make a decision in the end, but are much more free to change it.
Both Objective-C and Swift have (a form of) generics on board.
Re: APFS does not normalize Unicode filenames
#63More generally, once APFS is deployed users can legitimately end up with multiple files in the same folder whose names only differ in normalization. The initial message that starts this off seems to imply the opposite - instead, application developers should be normalizing the name before handing it to the filesystem. In that case, an application which allowed non-normalized naming would arguably have a bug.
Asking applications to enforce an OS-wide policy is asking for problems, even ignoring the existence of malware.
You wouldn't say "the OS doesn't check file attributes to see whether you are allowed to access a file; that's left to applications", either.
Re: APFS does not normalize Unicode filenames
#64Wouldn't this be seen as an issue in betas? I haven't seen anything indicating this is widespread so far? Why would that be, just not wide enough deployment yet?
Mabe most beta testers are based in english-speaking countries and countries where most people are used to stay away from non-english characters and never noticed the problem? I live in Sweden and still avoid using åäö in filenames because of old habits from DOS/Atari era.
It's sad when we give up to technological limitations!
They are our tools, not the other way around ;-)
Re: APFS does not normalize Unicode filenames
#65Earlier quoted context omitted.
How do you normalize arab or chinese in a meaningful way for people speaking these languages ?
You don't, there is only one unicode representation of those characters.
Interestingly that created its own annoyances for some since normalization occurs at a different stage.
Unicode and UTF-8 unify many Traditional & Simplified Chinese Characters as well as Japanese Kanji and Korean Hanja.
There were some extremely angry reactions to Han Character Unification[1], though efforts have been made to overcome many of the initial complaints.
[1] "Why Unicode Won’t Work on the Internet: Linguistic, Political, and Technical Limitations" http://www.hastingsresearch.com/net/04-unicode-limitations.s...
Re: APFS does not normalize Unicode filenames
#66The way iOS abstracts the filesystem away from user-view makes this less of an issue than it otherwise would be but still a good find by the author, as an aside surely I'm not the only one who thought of [1] when I read "APFS now treats all files as a bag of bytes on iOS" ;)
Re: APFS does not normalize Unicode filenames
#67Don't Mac apps already have to deal with network and FAT32 drives? Or does macOS already normalize those?
Re: APFS does not normalize Unicode filenames
#68I want to make the gzipped contents of my file it's name and leave the actual contents blank or whatever metadata. Thanks apfs!
Re: APFS does not normalize Unicode filenames
#69I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
> Things are much easier for the file system if it can just treat names as bags of bytes.
And much, much harder for applications if that "bag of bytes" is an invalid UTF-8 sequence. You will end up with an invalid string (or an exception), and trying to open that file will then fail.
I'd really hope that Apple checks that the filenames are valid UTF-8 as they otherwise can end up with very interesting security bugs.
Re: APFS does not normalize Unicode filenames
#70Earlier quoted context omitted.
This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's b…
> What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Show two files with apparently identical names. Is that so surprising? There are many many ways for two different Unicode strings to look visually identical (or near-identical) even if they aren't equivalent under normalization. For…
Even with normalization, you can just sprinkle an arbitrary selection of non breaking zero width space characters into a string and create an infinite variety of identical-looking but distinct codepoint sequences. Treating filenames as bundles of bytes is fine, but at that point it seems dangerous to assume that interpreting them as encoded strings is a valid way of distinguishing them.