Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

Post reply on HN
Print view View original

41–50 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#41

Post by [deleted] »

atoav wrote: post #11
tialaramex wrote: post #9

Earlier quoted context omitted.

In terms of what filenames are neither Windows nor Linux (I don't know for sure with MacOS but I doubt it) actually guarantee you any sort of characters . Linux filenames are a sequence of non-zero bytes (they might be ASCII, or at least UTF-8, they might be an old 8-bit charset, but they also might just be arbitrary non-zero bytes) and Windows file names are a sequence of non-zero 16-bit unsigned integers, which you…

This is what I found quite refreshing about Rust — instead of choosing one of the following: A) The programmer is a almighty god who knows everything, we just expose him to the raw thing B) The programmer is a immature toddler who cannot be trusted, so we handle things for them What Rust does is more among the lines of "you might already know this, but anyways here is a reminder that you, the programmer need to take…

[deleted]

Re: Unicode Normalization Forms: When ö ≠ ö

#43

Post by jrochkind1 »

misnome wrote: post #3

Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.

Well, precisely because if you don't normalize the filenames, ö ≠ ö. You could have two files with different filenames, `göteborg.txt` and `göteborg.txt`, and they are different files with different filenames.

Or you could have one file `göteborg.txt`, and when you try to ask for it as `göteborg.txt`, the system tells you "no file by that name".

Unicode normalization is the solution to this. And the unicode normalization algorithms are pretty good. The bug in this case is that the system did not apply unicode normalization consistently. It required a non-default config option to be turned on to do so? I don't really understand what's going on here, but it sounds like a bug in the system to me that this would be a non-default config option.

Dealing with the entire universe of human language is inherently complicated. But unicode gives us some actually pretty marvelous tools for doing it consistently and reasonably. But you still have to use them, and use them right, and with all software bugs are possible.

But I don't think you get fewer crazy edge cases by not normalizing at all. (In some cases you can even get security concerns, think about usernames and the risk of `jöhn` and `jöhn` being two different users...). I know that this is the choice some traditional/legacy OSs/file systems make, in order to keep pre-unicode-hegemony backwards compat. It has problems as well. I think the right choice for any greenfield possibilities is consistent unicode normalization, so `göteborg.txt` and `göteborg.txt` can't be two different files with two different filenames.

[btw I tried to actually use the two common different forms of ö in this text; I don't believe HN normalizes them so they should remain.]

Re: Unicode Normalization Forms: When ö ≠ ö

#44

Post by int_19h »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

You can already map Unicode ranges to "code pages" of sorts, so how would that help?

Thing is, people who are not linguists do want to mix languages. It's very common in some cultures to intersperse the native language with English. But even if not, if the language in question uses a non-Latin alphabet, there are often bits and pieces of data that have to be written down in Latin. So that "most of us" perspective is really "most of us in US and Western Europe", at best.

For domains and such, what I think is really needed is a new definition of string equality that boils down to "are people likely to consider these two the same?". So that would e.g. treat similarly-shaped Latin/Greek/Cyrillic letters the same.

Re: Unicode Normalization Forms: When ö ≠ ö

#45

Post by int_19h »

guerrilla wrote: post #32
B-Con wrote: post #29

Earlier quoted context omitted.

I think that in the ls->read workflow, Nextcloud shouldn't normalize the response from SMB and should issue back to SMB whatever SMB returned to Nextcloud.

According to Unicode, it should be allowed to and the SMB server should be able to handle it. That's kind of the point of normalization, they're meant to be done before all comparisons so that exactly this doesn't happen. Your suggestion is just premature optimization, i.e. eliminating a redundancy.

Unicode doesn't say anything about what "should be allowed to" with respect to an unrelated protocol. If the protocol says that filenames are sequences of 16-bit values that have to be compared one by one, then that's what it is.

Re: Unicode Normalization Forms: When ö ≠ ö

#46

Post by rzzzt »

dataflow wrote: post #28
rzzzt wrote: post #15

Earlier quoted context omitted.

Maintain a table that maps the original file name to random-generated one that doesn't hit these gotchas.

I'm afraid I don't follow. Who maintains this table and who consumes it? What if they're different entities? How do you prevent it from going out of sync with the file system when the user renames a file? Are you inventing your own file system here? How do you deal with existing file systems?

I assumed that you have a system where file management/synchronization happens strictly through a web interface, and files are not changed or renamed outside this system's knowledge. Under these preconditions, having such a mapping table frees the users from having to abide whatever restrictions the underlying file system places on valid file names.

Re: Unicode Normalization Forms: When ö ≠ ö

#47

Post by int_19h »

javajosh wrote: post #39

tl;dr - don't use crazy unicode characters in filenames, they can be problematic for non-trivial reasons (in this case because of unicode normalization on an smb mount.)

What's "crazy" about the letter? It's a standard letter of several European alphabets.

Re: Unicode Normalization Forms: When ö ≠ ö

#48

Post by dataflow »

rzzzt wrote: post #46
dataflow wrote: post #28

Earlier quoted context omitted.

I'm afraid I don't follow. Who maintains this table and who consumes it? What if they're different entities? How do you prevent it from going out of sync with the file system when the user renames a file? Are you inventing your own file system here? How do you deal with existing file systems?

I assumed that you have a system where file management/synchronization happens strictly through a web interface, and files are not changed or renamed outside this system's knowledge. Under these preconditions, having such a mapping table frees the users from having to abide whatever restrictions the underlying file system places on valid file names.

Oh I was talking about the general case from a programming standpoint. What do you do on a typical local filesystem?

The point I'm trying to get at being, you need to worry about the representation at multiple layers, not just at the bottom FS layer.

Re: Unicode Normalization Forms: When ö ≠ ö

#50

Post by mr_luc »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Heh, funny, I'm implementing this exact thing at the moment, oddly enough -- rather, implementing a security check that provides that same guarantee you mention, Mixed Script protections.

In Unicode spec terms, 'UTS 39 (Security)' contains the description of how to do this, mostly in section 5, and it relies on 'UTX 24 (Scripts)'.

It's more nuanced than your example but only slightly. If you replace "German" with "Japanese" you're talking about multiple scripts in the same 'writing system', but the spec provides files with the lists of 'sets of scripts' each character belongs to.

The way that the spec tells us to ensure that the word 'microsoft' isn't made up of fishy characters is that we just keep the intersection of each character's augmented script sets. If at the end, that intersection is empty, that's often fishy -- ie, there's no intersection between '{Latin}, {Cyrillic}'.

However, the spec allows the legit uses of writing systems that use more than one script; the lookup procedure outlined in the spec could give script sets like '{Jpan, Kore, Hani, Hanb}, {Jpan, Kana}' for two characters, and that intersection isn't empty; it'd give us the answer "Okay, this word is contained within the Japanese writing system".

Post reply on HN