Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

Post reply on HN
Print view View original

81–90 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#81

Post by rob_c »

DangitBobby wrote: post #71
jodrellblank wrote: post #14

Earlier quoted context omitted.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…

> word WORD Word and woRD as the same thing I don't know about anyone else, but I read WORD as someone yelling, Word as designating/specifying a "word" with some importance, and woRD as the mocking Spongebob meme. I absolutely don't read "case insensitive" and I don't think filesystems should either.

Have to agree. It's also usually only about 10 lines of code to support both insensitive and sensitive searching for those who can't read English that way.

Re: Unicode Normalization Forms: When ö ≠ ö

#82

Post by izacus »

LAC-Tech wrote: post #67
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Sprinkling English with foreign words is really, really common. I'm in New Zealand and people do it all the time. And even in the states, right? Don't want two different strings because someone writes an English sentence about how much they love jalapeño.

Think of just something simple like writing an immigrants name inside a sentence. It's kinda funny that people in SV, full of immigrants, never seem to think of putting their own or coworkers name in a String.

Re: Unicode Normalization Forms: When ö ≠ ö

#83

Post by rurban »

A filesystem accepting only NFD should be filed as bug. They can normalize it internally to NFD, as Apples previous HFS+ did.

But even worse than that is Python's NFKC, which normalizes ℌ to H and so on. The recommended normalizations are NFC for offline normalization (like in compiled languages and databases) and NFD for online, where speed trump's space. unicode.org talking that much about NFKC was a big mistake. NFKC is crazy and doesn't even roundtrip. The whole TR31 XID_Start/Continue sets are mostly because of NFKC issues, not so about stability. But people bought it for its stability argument.

I'm just writing a library and linter for such issues: https://github.com/rurban/libu8ident

Also note that C++23 will most likely enforce NFC identifiers only. Same problem as with this filesystem. My implementation was to accept all normal. forms and store it internally and in the object files as NFC. The C ABI should declare it also. Currently they don't care as much as Linux filesystems: Nada. Identifiers being unidentifiable

Re: Unicode Normalization Forms: When ö ≠ ö

#84

Post by Nitramp »

drdaeman wrote: post #55
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> but for most of us we would be better off That's simple - it is provably wrong. While relatively uncommon there are plenty of examples that would contradict this statement. And it's not about being able to encode the Rosetta Stone - non-scientists mix languages all the time, from Carmina Burana to Blinkenlights. They even make meaningful portmanteau words and write them with characters from multiple unrelated writi…

I'm intrigued, what's заshitано?

Re: Unicode Normalization Forms: When ö ≠ ö

#85

Post by DangitBobby »

jodrellblank wrote: post #77
DangitBobby wrote: post #71

Earlier quoted context omitted.

> word WORD Word and woRD as the same thing I don't know about anyone else, but I read WORD as someone yelling, Word as designating/specifying a "word" with some importance, and woRD as the mocking Spongebob meme. I absolutely don't read "case insensitive" and I don't think filesystems should either.

You read DOG as someone yelling ‘dog’, not as a different word to ‘dog’. And Dog as a significant dog, not a significant something else. Imagine if you could only search for ‘dog’ if you had to specify whether the author yelled it or not before you could find it.

It sounds like you're saying that cases should matter in some ways but not in others, which I take no issue with.

Re: Unicode Normalization Forms: When ö ≠ ö

#86

Post by [deleted] »

danudey wrote: post #58
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…

[deleted]

Re: Unicode Normalization Forms: When ö ≠ ö

#87

Post by BlueTemplar »

danudey wrote: post #58
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…

Funnily base64 suffers from a related issue that the likes of base58 correct : l and I or O and 0 looking similar or even identical depending on the font !

Re: Unicode Normalization Forms: When ö ≠ ö

#88

Post by smitty1e »

danudey wrote: post #58
hinkley wrote: post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

In our Jenkins system, we have remote build nodes return data back to the primary node via environment variable-style formatted files (e.g. FOO=bar), so when I had to send back a bunch of arbitrary multi-line textual data, I decided to base64 encode it. Simple enough. On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any specia…

We had to ETL .csv data that must have originated in SQLServer.

The utf-16 fact about Windows was apparently unknown to my predecessor.

Who wrote some nasty c-language binary to copy the data, knock the upper byte off of each character ahead, and save the now ASCII text to a new file of the mysql load.

The encoding='utf-16' argument was all that was needed.

For want of a nail. . .

Re: Unicode Normalization Forms: When ö ≠ ö

#89

Post by smitty1e »

wongarsu wrote: post #76
cgriswald wrote: post #72

Earlier quoted context omitted.

I'm not a linguist and that will probably be readily apparent. The word jalapeño leaves me wondering how distinct a boundary a language can possess or how one can sort out which language an individual word belongs to outside the context of the rest of the text or speech. In English, jalapeño is correctly spelled with or without the eñe (and AFAIK the letter doesn't have a name in English, you have to use the Spanish…

Words like angst or ersatz are English words borrowed from German. The German words are written identically (except capitalisation), but the meaning of the English word is much more specific than the German "original". Meanwhile the word "Blitz" has completely district meanings in English and German. In English it's a sudden concerted effort, in German it's lightning. Despite the English word originating from German,…

Isn't the etymology of 'blitz' in English due more to 'blitzkrieg'?

Re: Unicode Normalization Forms: When ö ≠ ö

#90

Post by dagmx »

drpixie wrote: post #52
int_19h wrote: post #47

Earlier quoted context omitted.

What's "crazy" about the letter? It's a standard letter of several European alphabets.

Nothing crazy about the "letter", but it is crazy that there are multiple different ways to encode the "letter".

A user wouldn't know that there are multiple ways to encode a given character unless they're experienced with Unicode.

Additionally there are (iirc) multiple ways to encode characters even in the ASCII set.

This is purely a failing in consistent normalization schemes.

Post reply on HN