Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

Post reply on HN
Print view View original

51–60 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#51

Post by guerrilla »

int_19h wrote: post #45
guerrilla wrote: post #32

Earlier quoted context omitted.

According to Unicode, it should be allowed to and the SMB server should be able to handle it. That's kind of the point of normalization, they're meant to be done before all comparisons so that exactly this doesn't happen. Your suggestion is just premature optimization, i.e. eliminating a redundancy.

Unicode doesn't say anything about what "should be allowed to" with respect to an unrelated protocol. If the protocol says that filenames are sequences of 16-bit values that have to be compared one by one, then that's what it is.

It does say that if comparisons are being made then... and comparisons are being made, so yes, it does.

Re: Unicode Normalization Forms: When ö ≠ ö

#52

Post by drpixie »

int_19h wrote: post #47
javajosh wrote: post #39

tl;dr - don't use crazy unicode characters in filenames, they can be problematic for non-trivial reasons (in this case because of unicode normalization on an smb mount.)

What's "crazy" about the letter? It's a standard letter of several European alphabets.

Nothing crazy about the "letter", but it is crazy that there are multiple different ways to encode the "letter".

Re: Unicode Normalization Forms: When ö ≠ ö

#53

Post by Lammy »

justaguy37 wrote: post #17
laurent92 wrote: post #12

Earlier quoted context omitted.

So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? And John.McCauley@yahoo.fr and john.mccauley@yahoo.fr being attributed to two different people ;)

can we just say no to capital letters? (or lowercase?) do capital letters have a good enough usage case to justify their continued existence?

Fun fact: The Apple Ⅱ and Ⅱ+ originally only did upper-case, and it was very popular to add a Shift Key / lower-case mod via one of the gamepad buttons: https://web.archive.org/web/20010212094858/http://home.swbel...

Re: Unicode Normalization Forms: When ö ≠ ö

#54

Post by jrochkind1 »

int_19h wrote: post #44
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

You can already map Unicode ranges to "code pages" of sorts, so how would that help? Thing is, people who are not linguists do want to mix languages. It's very common in some cultures to intersperse the native language with English. But even if not, if the language in question uses a non-Latin alphabet, there are often bits and pieces of data that have to be written down in Latin. So that "most of us" perspective is…

Oh, you can do far more than "code pages of sorts". Unicode has a variety of metadata available about each codepoint. The things that are "code pages of sorts" are maybe "block" (for ö "Latin-1 Supplement"), and "plane" (for ö it's "Basic Multilingual Plane"), but those are really mostly administrative and probably not what want.

But you also have "Script" (for ö "Latin). Some characters belong to more than one script though. Unicode will tell you that.

Unicode also has a variety of algorithms available already written. One of the most relevant ones here is... normalization. To compare two strings in the broadest semantic sense of "are people likely to consider these the same", you want want a "compatibility" normalization. NFKC or NFKD. They will for instance make `1` and `¹`[superscript] the same, which is definitely one kind of "consider these the same" -- very useful for, say, a search index.

That won't be iron-clad, but that will be better than trying to role your own algorithm involving looking at character metadata yourself! But it won't get you past intentional attacks using "look-alike" characters that are actually different semantically but look similar/indistinguishable depending on font. The trick is "consider these the same" really, it turns out, depends on context and purpose, it's not always the same.

Unicode also has a variety of useful guides as part of the standard, including the guide to normalization https://unicode.org/reports/tr15/ and some guides related to security (such as https://unicode.org/reports/tr36/ and http://unicode.org/reports/tr39/), all of which are relevant to this concern, and suggest approaches and algorithms.

Unicode has a LOT of very clever stuff in it to handle the inherently complicated problem of dealing with the entire universe of global languages that Unicode makes possible. It pays to spend some time with em.

Re: Unicode Normalization Forms: When ö ≠ ö

#55

Post by drdaeman »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> but for most of us we would be better off

That's simple - it is provably wrong. While relatively uncommon there are plenty of examples that would contradict this statement. And it's not about being able to encode the Rosetta Stone - non-scientists mix languages all the time, from Carmina Burana to Blinkenlights. They even make meaningful portmanteau words and write them with characters from multiple unrelated writing systems, like "заshitано" (see - Latin and Cyrillic scripts in the same single word!)

Re: Unicode Normalization Forms: When ö ≠ ö

#56

Post by david-gpu »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us we would be better off being able to a assert that a piece of text was German, or sanskrit, not a jumble of both.

Presumably the person who wrote it speaks a single language.

Just because something is not useful to them, it doesn't mean it is not useful in general. There are millions of polyglots as well as documents that include words and names in multiple scripts.

Re: Unicode Normalization Forms: When ö ≠ ö

#57

Post by GlitchMr »

chrismorgan wrote: post #33
lilyball wrote: post #24

Earlier quoted context omitted.

By “proprietary variant” you mean “publicly documented variant” which IIRC is just the normalization tables frozen in time from an early version of Unicode (the idea being that updating your OS shouldn’t change the rules about what filenames are valid). As for APFS, it ~~doesn’t~~didn’t normalize but I believe it still requires UTF-8. And the OS will normalize filenames at a higher level. EDIT: they added native norm…

Normalisation is expressly done with the composition of version 3.1 for compatibility: see https://www.unicode.org/reports/tr15/#Versioning >. IF that’s what HFS+ does, then “proprietary variant” is wrong. And if not, I’m curious what it does differently. (On the use of version 3.1, note that in practice version 3.2 is used, correcting one typo: see https://www.unicode.org/versions/corrigendum3.html >.) I find a few…

The `filename-sanitizer` library you have linked has the following comment.

                # FIXME: improve HFS+ handling, because it does not use the standard NFD. It's
                # close, but it's not exactly the same thing.
                'hfs+': (255, 'characters', 'utf-16', 'NFD'),
I wonder what does that mean...

Re: Unicode Normalization Forms: When ö ≠ ö

#58

Post by danudey »

hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough.

On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any special characters in our commit messages).

On Windows systems... oh god. The system treats all text as UTF-16 with whatever byte order, and it took me ages to figure out how to get it to convert the data to UTF-8 before encoding it. Eventually it started working, and it worked for a while until it didn't for whatever reason. I ended up tearing out all the code and just encoding the UTF-16 in base64 and then processing that into UTF-8 on the master where I had access to much saner tools.

Generally speaking, "Unicode" works great in most cases, but when you're dealing with systems with weird or unusual encoding habits, like Windows using UTF-16 or MySQL's "utf8" being limited to three bytes per unicode character instead of four, everything goes out the window and it's the wild west all over again.

Re: Unicode Normalization Forms: When ö ≠ ö

#60

Post by nieve »

jrochkind1 wrote: post #43
misnome wrote: post #3

Why isn’t the answer just “Don’t unicode normalise the file name”? I thought the generally recommended way to deal with file names is to treat as a block of bytes (to the extent that e.g. rust has an entirely separate string type for OS provided strings), or just to allow direct encoding/decoding but not normalisation or alteration.

Well, precisely because if you don't normalize the filenames, ö ≠ ö. You could have two files with different filenames, `göteborg.txt` and `göteborg.txt`, and they are different files with different filenames. Or you could have one file `göteborg.txt`, and when you try to ask for it as `göteborg.txt`, the system tells you "no file by that name". Unicode normalization is the solution to this. And the unicode normal…

It looks like instead of the config option switching everything to use the same normalization it keeps a second copy of the name in a database to compare to. What a horrible kludge, I wonder how they even got into this situation of using different normalization in different parts of the system?
Post reply on HN