141–148 of 148 posts
I'm not sure if normalization is good idea (generally because Unicode is complex beast and moving that complexity inside a kernel should be carefully weighted), but I'm sure that it doesn't solve any real problem. Characters "A" and "А" looks identical, unless you're missing Cyrillic font, but they won't be normalized, because they are completely different characters. There are many more other visually identical stri…
Earlier quoted context omitted.
Canonically equivalent Unicode sequences look the same to machines. Memcmp disagrees, as do the default equality operators of most programming languages in existence.
Sure, but normalisation can nonetheless happen automatically and implicitly in many places.
Earlier quoted context omitted.
The problem exists already on the visual level, since there are pairs of distinct Unicode characters that look very much alike. I think the only thing you can really do about that is restricting to ASCII minus control characters. Probably many programmers would be willing to accept that, but non-technical users not. The next best thing would be to enforce a canonical (code-point?) encoding. But given the complexities…
> The problem exists already on the visual level, since there are pairs of distinct Unicode characters that look very much alike. Yes, but software doesn't consider them equivalent.
I agree that this is a good change. Unicode, normalisation, character encodings, etc. should really be handled at the presentation layer, and everything below that just treats filenames as sequences of bytes, perhaps with one or two exceptions like '/' and \0. It is interesting to consider a theoretical system in which paths are represented in 0-terminated count-length format (e.g. "foo/bar/baz/myfile.txt" would be "…
No. Ever heard about http://websec.github.io/unicode-security-guide/ Identifiers should be identifiable. If a filename is encoded in utf8, it needs to be normalized. To the canonical form of course not the crazy Python 3 or Apple idea of NFD. Which is also slower. If it's encoded as bytes you get garbage in - garbage out. There's much more to consider, but unfortunately you cannot restrict a directory to forbid mixed…
That's fortunate, not unfortunate. I want the files corresponding to documents to be named according to the titles of those documents, and said titles often do mix Latin and Cyrillic scripts in a way that produces "confusables". So as a generic rule for all filenames, it's a no-go - it will significantly affect perfectly legitimate use cases.
Earlier quoted context omitted.
> Which part of it is relevant to pathnames? All parts regarding identifiers and names, if you see pathnames as publicly identifiable information. See below. > In what circumstances does Python 3 normalize pathnames? None. Python 3 normalizes identifiers to the shorter non-canonical NFKC form. Apple HFS+ normalizes pathnames to the longer D canonical form. Which is faster but takes more space. Usually space is more i…
> to the shorter non-canonical NFKC form What definition or sense of 'canonical' do you mean? NFKC stands for "Compatibility Decomposition, followed by Canonical Composition", so it's canonical in some sense, right?
Earlier quoted context omitted.
I think, the problem with Unicode normalization is, input methods are (well, conceptually, at least) meant to produce text, not binary. If filesystems are using binary data for filenames, there can be a case when is really no way to address a file by typing its name, even if you can type in that language. This isn't an issue for case sensitivity or alike.
> If filesystems are using binary data for filenames, there can be a case when is really no way to address a file by typing its name, even if you can type in that language You could do that trivially in UNIX since forever.
I would've actually preferred for identifiers to be Unicode text strings.
Earlier quoted context omitted.
I don't know the technical details but the UTF-8 from a Mac is different from UTF-8 on Linux. Copy "hällö" from Mac to Linux over samba -> works fine. scp or rsync that same file again from Mac to Linux (without special conversion options) -> you will have two "hällö" there. The first one is recognized by Samba, the second one is invisible.
No, UTF-8 is the same everywhere. What you're confused about is the fact that for a lot of strings there are multiple different unicode scalar value sequences that represent the same user-visible string. This is the difference between composed and decomposed characters. For example, é can be represented either as a single scalar value that represents e-with-acute, or as two scalar values, a regular e followed by a co…
Earlier quoted context omitted.
No, UTF-8 is the same everywhere. What you're confused about is the fact that for a lot of strings there are multiple different unicode scalar value sequences that represent the same user-visible string. This is the difference between composed and decomposed characters. For example, é can be represented either as a single scalar value that represents e-with-acute, or as two scalar values, a regular e followed by a co…
Well one of this (the Apple one) is clearly wrong because Samba (which is not a niche software) does not work with it and this is critical for me. Just saying I'm not the only one who is confused ;)
But this doesn't make it wrong. You can reproduce this exact behavior with other systems too simply by changing the local filename to be decomposed. The only Mac-specific part here is that HFS+ will automatically decompose the filename.