41–50 of 148 posts
Wouldn't this be seen as an issue in betas? I haven't seen anything indicating this is widespread so far? Why would that be, just not wide enough deployment yet?
Earlier quoted context omitted.
This feels in some sense like punting the problem. What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Surface the normalization form the user? Uh, no... The more fundamental question is whether filenames are under control of the user or the system. The answer today is "both": there's b…
> What exactly should the presentation layer do when presenting two files, where the first is named with a precomposed character sequence, and the other has the same name but decomposed? Show two files with apparently identical names. Is that so surprising? There are many many ways for two different Unicode strings to look visually identical (or near-identical) even if they aren't equivalent under normalization. For…
http://www.sami-lehtinen.net/blog/linux-windows-ntfs-differe...
Unicode isn't required to mess up things. Here's what baffled me for a while with NTFS. I'm pretty sure these issues are well known. http://www.sami-lehtinen.net/blog/linux-windows-ntfs-differe...
I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
I'm with David Wheeler: https://www.dwheeler.com/essays/fixing-unix-linux-filenames....
We need to limit filenames for the good of the entire system and the whole community. Filenames as byte strings may sound good, but nobody ever thinks of the costs and the scant benefits.
Is APFS still using Apple's style UTF-8 for e.g. Umlauts? I had a lot of trouble with rsync and also Samba later (filenames and folders hidden) when I discovered that Umlauts on HFS are different than Umlauts on e.g. Ext4.
What do you mean by "different"? Umlauts on HFS+ are still Umlauts anywhere else. The only real oddity of HFS+ (beyond the fact that it does normalization at all) is that it's not using NFD, it's using a variant of NFD based on an old version of Unicode (it has to be this way because the normalization tables must be immutable or there's compatibility issues when reading drives written to from different versions of th…
I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
No. Ever heard about http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed…
Which part of it is relevant to pathnames?
> To the canonical form of course
Do you mean NFC?
> not the crazy Python 3 or Apple idea of NFD.
In what circumstances does Python 3 normalize pathnames?
> If it's encoded as bytes you get garbage in - garbage out.
What do you mean by "encoded as bytes"?
> http://perl11.org/blog/unicode-identifiers.html
This is about programming language identifies, not about pathnames, which are very different beasts.
I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
It's a good idea until you end up with two files that have the "same" name (eg. Amélie.jpg and Amélie.jpg) because one uses decomposed characters (U+0065 and U+0301) and the other one uses a single character (U+00E9). If the difference is not visible in your browser (it shouldn't), try copy-pasting those two filenames in a text editor, one of them is 10 characters long and one of them is 11 characters long.
I think the only thing you can really do about that is restricting to ASCII minus control characters. Probably many programmers would be willing to accept that, but non-technical users not.
The next best thing would be to enforce a canonical (code-point?) encoding. But given the complexities of Unicode that will get us only so far...
Earlier quoted context omitted.
No. Ever heard about http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed…
> No. Ever heard about http://websec.github.io/unicode-security-guide/ Which part of it is relevant to pathnames? > To the canonical form of course Do you mean NFC? > not the crazy Python 3 or Apple idea of NFD. In what circumstances does Python 3 normalize pathnames? > If it's encoded as bytes you get garbage in - garbage out. What do you mean by "encoded as bytes"? > http://perl11.org/blog/unicode-identifiers.html…
All parts regarding identifiers and names, if you see pathnames as publicly identifiable information. See below.
> In what circumstances does Python 3 normalize pathnames?
None. Python 3 normalizes identifiers to the shorter non-canonical NFKC form. Apple HFS+ normalizes pathnames to the longer D canonical form. Which is faster but takes more space. Usually space is more important than CPU.
> What do you mean by "encoded as bytes"?
What everyone else means on byte encodings: 1-1 mapping of bytes, without any further knowledge of the encoding.
> This is about programming language identifies, not about pathnames, which are very different beasts.
Partially. pathnames can also be argued to be identifiers, similar to domain names, email names, user names, language identifiers. Apple does so. Ask plan9 what they think about pathname semantics. Not everyone is in the garbage in - garbage out camp. Many systems do encode pathnames and have character restrictions. Esp. important is the popular / spoof, which people from the garbage camp don't care about. Julia is recent proponent of the garbage camp, btw.