Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

91–100 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#91
When Unicode adopted normalization it went off the rails into mudville. Then, determined to make a mockery of its purpose, it adopted semantic meanings, fonts, and then started inventing all sorts of new characters.

"If you vote for my nutburger glyph my kid drew for a kindergarten assignment, I'll vote for the chicken scratching you noticed in the barnyard dust."

Re: Unicode Normalization Forms: When ö ≠ ö

#92
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Sprinkling English with foreign words is really, really common. I'm in New Zealand and people do it all the time. And even in the states, right? Don't want two different strings because someone writes an English sentence about how much they love jalapeño.

My grandfather's thesis was auf Deutsch and is sprinkled with French and Latin words.

Re: Unicode Normalization Forms: When ö ≠ ö

#93

Earlier quoted context omitted.

I'm not a linguist and that will probably be readily apparent. The word jalapeño leaves me wondering how distinct a boundary a language can possess or how one can sort out which language an individual word belongs to outside the context of the rest of the text or speech. In English, jalapeño is correctly spelled with or without the eñe (and AFAIK the letter doesn't have a name in English, you have to use the Spanish…

Words like angst or ersatz are English words borrowed from German. The German words are written identically (except capitalisation), but the meaning of the English word is much more specific than the German "original". Meanwhile the word "Blitz" has completely district meanings in English and German. In English it's a sudden concerted effort, in German it's lightning. Despite the English word originating from German,…

German has the word blitzschnell which means really fast (as fast a bolt). So in a way the english meaning of blitz still fits your description.

Re: Unicode Normalization Forms: When ö ≠ ö

#94
post #58

Earlier quoted context omitted.

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…

Funnily base64 suffers from a related issue that the likes of base58 correct : l and I or O and 0 looking similar or even identical depending on the font !

Why does that matter? When would a human need to read and comprehend base64 encoded data?

Re: Unicode Normalization Forms: When ö ≠ ö

#95
post #35
post #26

Earlier quoted context omitted.

> So you’re fine with ~/Downloads and ~/downloads coexisting as entirely separate directories? Case (in)sensitivity for filenames is a non-issue in my experience. Never had problems with either convention. As for emails, I do think insensitivity was the right choice.

The RFC states that email addresses are case sensitive. The local-part of a mailbox MUST BE treated as case sensitive. Section 2.4 RFC 2821, https://www.ietf.org/rfc/rfc2821.txt

Ah interesting. I guess the case insensitivity (for incoming email) is a decision of the popular services then, like gmails decision to consider johndoe equivalent to john.doe.

Re: Unicode Normalization Forms: When ö ≠ ö

#96
post #52
post #47

Earlier quoted context omitted.

What's "crazy" about the letter? It's a standard letter of several European alphabets.

Nothing crazy about the "letter", but it is crazy that there are multiple different ways to encode the "letter".

So, no combining characters? Ok, even if you rule out Latin characters with accents and only use code points that consider the character with accent a single entity... you still have world languages which need combining in order to work, which means you can't really escape multiple encodings of the same "graphemes" (these languages don't exactly have "letters" like ASCII does).

Re: Unicode Normalization Forms: When ö ≠ ö

#97
post #58

Earlier quoted context omitted.

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…

We had to ETL .csv data that must have originated in SQLServer. The utf-16 fact about Windows was apparently unknown to my predecessor. Who wrote some nasty c-language binary to copy the data, knock the upper byte off of each character ahead, and save the now ASCII text to a new file of the mysql load. The encoding='utf-16' argument was all that was needed. For want of a nail. . .

I've had to fix this before. A co-worker working with data from a 3rd party supplier had gone "Oh this input data is mangled with stray zero bytes, I'll fix that" and of course that destroys any non-ASCII inputs, eventually I'm told that sometimes the import fails, I investigate, and I realise the "mangled" input is just UTF-16 encoded, conditionally remove the "strip zero bytes" hack and tell the decoder it's UTF-16 and it just works correctly.

The "maybe strip null bytes" code lived for years "just in case" after I fixed that because people couldn't believe that's all that was ever "wrong" with the data.

Re: Unicode Normalization Forms: When ö ≠ ö

#98
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Of course living in denial makes it easy to ignore harsh realities. Unfortunately for them, humans don't work that way. Things aren't gonna spontaneously change just to make their life easier. The software adapts to us, not the other way around. People complain about the complexities of dates and times but they still make every effort to get it right because it matters.

If a programming language allows text processing but can't even properly compare unicode text, it is buggy and needs to be fixed. If an operating system can't deal with unicode, it's buggy and needs to be fixed.

Re: Unicode Normalization Forms: When ö ≠ ö

#99
post #58
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…

The perils of valuing backwards compatibility above all else… imagine having to use UTF16 in this day and age. Happy 2022!

Re: Unicode Normalization Forms: When ö ≠ ö

#100

Earlier quoted context omitted.

Normalisation is expressly done with the composition of version 3.1 for compatibility: see https://www.unicode.org/reports/tr15/#Versioning >. IF that’s what HFS+ does, then “proprietary variant” is wrong. And if not, I’m curious what it does differently. (On the use of version 3.1, note that in practice version 3.2 is used, correcting one typo: see https://www.unicode.org/versions/corrigendum3.html >.) I find a few…

The `filename-sanitizer` library you have linked has the following comment. # FIXME: improve HFS+ handling, because it does not use the standard NFD. It's # close, but it's not exactly the same thing. 'hfs+': (255, 'characters', 'utf-16', 'NFD'), I wonder what does that mean...

The technote linked by the parent has a note saying

> The characters with codes in the range u+2000 through u+2FFF are punctuation, symbols, dingbats, arrows, box drawing, etc. The u+24xx block, for example, has single characters for things like "(a)". The characters in this range are not fully decomposed; they are left unchanged in HFS Plus strings. This allows strings in Mac OS encodings to be converted to Unicode and back without loss of information. This is not unnatural since a user would not necessarily expect a dingbat "(a)" to be equivalent to the three character sequence "(", "a", ")" in a file name.

> The characters in the range u+F900 through u+FAFF are CJK compatibility ideographs, and are not decomposed in HFS Plus strings.

The bit about the u+24xx block is misleading, the decomposition of the characters I looked at there (such as ⒜) are compatibility canonicalizations. However the CJK compatibility ideographs is a working example. U+F902 (車) decomposes to U+8ECA (車) regardless of normalization form but the technote says these must not be decomposed.

Post reply on HN