Most formats (including XML) require data to be normalized to NFC.
Unicode Normalization Forms: When ö ≠ ö
31–40 of 144 posts
Re: Unicode Normalization Forms: When ö ≠ ö
#32> But here, normalization caused this issue. Nope, the lack of normalization on both accounts by the SMB server caused the issue. It could have normalized before emitting but it definitely should have normalized on receiving for comparison.
I think that in the ls->read workflow, Nextcloud shouldn't normalize the response from SMB and should issue back to SMB whatever SMB returned to Nextcloud.
According to Unicode, it should be allowed to and the SMB server should be able to handle it. That's kind of the point of normalization, they're meant to be done before all comparisons so that exactly this doesn't happen. Your suggestion is just premature optimization, i.e. eliminating a redundancy.
Re: Unicode Normalization Forms: When ö ≠ ö
#33Earlier quoted context omitted.
Filenames in HFS+ filesystem (an old filesystem used by Mac OS X) are normalized with a proprietary variant of NFD - this is a filesystem feature. APFS removed this feature.
By “proprietary variant” you mean “publicly documented variant” which IIRC is just the normalization tables frozen in time from an early version of Unicode (the idea being that updating your OS shouldn’t change the rules about what filenames are valid). As for APFS, it ~~doesn’t~~didn’t normalize but I believe it still requires UTF-8. And the OS will normalize filenames at a higher level. EDIT: they added native norm…
Normalisation is expressly done with the composition of version 3.1 for compatibility: see https://www.unicode.org/reports/tr15/#Versioning>. IF that’s what HFS+ does, then “proprietary variant” is wrong. And if not, I’m curious what it does differently.
(On the use of version 3.1, note that in practice version 3.2 is used, correcting one typo: see https://www.unicode.org/versions/corrigendum3.html>.)
I find a few references to it being slightly different, but not one of them actually says what’s different; Wikipedia is the only one with a citation (https://en.wikipedia.org/wiki/HFS_Plus>: “and normalized to a form very nearly the same as Unicode Normalization Form D (NFD)[12]”), and that citation says it’s UAX #15 NFD, no deviations. One library that handles HFS+ differently switches to UCD 3.2.0 for HFS+ https://github.com/ksze/filename-sanitizer/blob/e990e963dc5b...>, but my impression from UAX #15 is that this should be superfluous, not actually changing anything. (Why is UCD 3.2.0 still around there? Probably because IDNA 2003 needs it: https://bugs.python.org/issue42157#msg379674>.)
Update: https://developer.apple.com/library/archive/technotes/tn/tn1... has actual technical information, but the table in question doesn’t show Unicode version changes like they claim it does, so I dunno. Looks like maybe from macOS 10.3 it’s exactly UAX #15, but 8.1–10.2 was a precursor? I’m fuzzy on where the normalisation actually happens, anyway.
Re: Unicode Normalization Forms: When ö ≠ ö
#34Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.
In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters . Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you…
macOS is interesting: some APIs normalize filenames while others don't. And it causes some very interesting bugs.
One example is when you submit a file in Safari it doesn't normalize the file name while js file.name does.
Re: Unicode Normalization Forms: When ö ≠ ö
#35Earlier quoted context omitted.
So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)
> So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? Case (in)sensitivity for filenames is a non-issue in my experience. Never had problems with either convention. As for emails, I do think insensitivity was the right choice.
The RFC states that email addresses are case sensitive.
The local-part of a mailbox MUST BE treated as case sensitive.
Section 2.4 RFC 2821, https://www.ietf.org/rfc/rfc2821.txt
Re: Unicode Normalization Forms: When ö ≠ ö
#36Well, if 7-bit US ASCII was good enough for our Lord, it is good enough for me ;-)
Re: Unicode Normalization Forms: When ö ≠ ö
#37Most formats (including XML) require data to be normalized to NFC.
Can you point me to a single format that actually requires NFC? Most things either make no comment or just express preferences, though I’m confident there will be some somewhere.
XML does not require normalisation: per https://www.w3.org/TR/xml11/#sec-normalization-checking>, XML data SHOULD be fully normalised, but MUST NOT be transformed by processors; in other words, it’s a dead letter “SHOULD”, and no one actually cares, just like almost everything else.
Re: Unicode Normalization Forms: When ö ≠ ö
#38Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.
Falls over on the fact that I don’t want to be able to write these two files in the same dir. if I write file ö1.txt and ö1.txt then I want to be warned that the file exists even of the encoding is different when I use two different apps but try to write the same file.
The same applies for a.txt and A.txt on case insensitive file systems (as someone pointed out the most common desktop file systems are).
Re: Unicode Normalization Forms: When ö ≠ ö
#39tl;dr - don't use crazy unicode characters in filenames, they can be problematic for non-trivial reasons (in this case because of unicode normalization on an smb mount.)
Re: Unicode Normalization Forms: When ö ≠ ö
#40Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters.
Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us we would be better off being able to a assert that a piece of text was German, or sanskrit, not a jumble of both. It's been living rent free in my head for almost two decades and I can't agree with it, nor can I laugh it off.
It might have been better if the 'code pages' idea was refined instead of eliminated (that is, the string uses one or more code pages, not the process). I don't know what the right answer is, but I know Every X is a Y almost always gets us into trouble.