Technically the presentation-layer problem existed already with things like legacy path separators, making the Finder tell lies in the presence of colons or slashes. I suspect that normalization differences will be a little like telling two files apart when one has a trailing space, or hidden file extensions; there will have to be some distinction but maybe no easy answer.
APFS does not normalize Unicode filenames
31–40 of 148 posts
Re: APFS does not normalize Unicode filenames
#32Linus Thorvalds will be happy to hear that http://www.cio.com/article/2868393/linus-torvalds-apples-hfs...
Re: APFS does not normalize Unicode filenames
#33I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's b…
> What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed?
Show two files with apparently identical names. Is that so surprising? There are many many ways for two different Unicode strings to look visually identical (or near-identical) even if they aren't equivalent under normalization. For that matter, even if you were limited to plain keyboard ASCII, you could have two really long filenames that only differ in one character, which are effectively indistinguishable without massive hair-pulling. If you want to be robust, there's no way around having means to identify files other than names.
(I agree with your sentiment about tagging.)
Re: APFS does not normalize Unicode filenames
#34I've been very excited about ReFS -- a real "modern" Filesystem that leaves legacy issues behind. We've been using it for large storage systems, and am hoping it will become a viable solution for everything soon. It solves most of these issues.
If Microsoft open source it with a free patent license it might be useful. Until that it is just another proprietary FS locked to a single OS.
Re: APFS does not normalize Unicode filenames
#35I've been very excited about ReFS -- a real "modern" Filesystem that leaves legacy issues behind. We've been using it for large storage systems, and am hoping it will become a viable solution for everything soon. It solves most of these issues.
If Microsoft open source it with a free patent license it might be useful. Until that it is just another proprietary FS locked to a single OS.
Like APFS?
Re: APFS does not normalize Unicode filenames
#36Is APFS still using Apple's style UTF-8 for e.g. Umlauts? I had a lot of trouble with rsync and also Samba later (filenames and folders hidden) when I discovered that Umlauts on HFS are different than Umlauts on e.g. Ext4.
Do you mean composed versus decomposed? Utf-8 allows both. Unfortunately this article doesn't explain it very well, here's a better link.
https://docs.syncthing.net/advanced/folder-autonormalize.htm...
Re: APFS does not normalize Unicode filenames
#37For anyone else wondering:
Re: APFS does not normalize Unicode filenames
#38Earlier quoted context omitted.
This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's b…
> What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Show two files with apparently identical names. Is that so surprising? There are many many ways for two different Unicode strings to look visually identical (or near-identical) even if they aren't equivalent under normalization. For…
You are talking about confusables.
Unfortunately normalization cannot handle this. Only certain apps treat confusables as confusables: email, web browsers, dns services, usually via libidn.
Re: APFS does not normalize Unicode filenames
#39I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
No. Ever heard about http://websec.github.io/unicode-security-guide/
Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out.
There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed scripts or confusables. I summarized a few problems at http://perl11.org/blog/unicode-identifiers.html
Re: APFS does not normalize Unicode filenames
#40I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
How do you normalize arab or chinese in a meaningful way for people speaking these languages ?