Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

111–120 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#111
post #74
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Reminds me of the good old days with EUC-KR, KSC 5601, and all those different encoding schemes I've successfully repressed in my memory for years. Yes, you could probably assert that a piece of string was either Korean or English but never anything else... because the system was incapable of representing it. I'm not exactly sure how a code page is supposed to help us here. Developers have trouble supporting multiple…

I'd guess a standardized codepage marker like a "start of CP[932]” is going to be necessary CP[1252] at each CP switches but it might be just a necessity. Han unification is a well known problem to Far Eastern but Unicode normalization problem is basically the same as that.

Re: Unicode Normalization Forms: When ö ≠ ö

#112
post #44

Earlier quoted context omitted.

You can already map Unicode ranges to "code pages" of sorts, so how would that help? Thing is, people who are not linguists do want to mix languages. It's very common in some cultures to intersperse the native language with English. But even if not, if the language in question uses a non-Latin alphabet, there are often bits and pieces of data that have to be written down in Latin. So that "most of us" perspective is…

Yeah, Greek alphabet is used a lot in sciences. It's really annoying that we're only starting to get proper support now . (Including on keyboards : http://norme-azerty.fr/en/ )

By far the most common use case these days is when an URL has to be mentioned in an otherwise non-Latin text.

Re: Unicode Normalization Forms: When ö ≠ ö

#113
post #8

Earlier quoted context omitted.

Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…

Case insensitivity and "what human brains do" becomes incredibly complicated outside of English. There are also many other things which human brains recognize as the same thing but would be unreasonable to implement in filesystems.

In Japanese, くるま, クルマ, and 車 are all the same word (the first two are the phonetic spelling "kuruma", the later is the Chinese character). However in order to know that 車 is read くるま you need to be a native Japanese speaker (or have a dictionary) -- should filesystems have dictionaries to match what a human would think? Search engines that support Japanese have to handle this to some degree, but I humbly suggest implement Google Search's language handling code into a filesystem would be an ill-advised decision.

If you wanted to implement the most minimal version of this you would map between katakana and hiragana, but that means you'll need to do this for other languages. For instance, Serbian. Serbian uses two scripts (both of which have upper and lower case forms) and any native Serbian speaker would see "tuđa ljuta paprika" and "туђа љута паприка" as the same text (note that lj became љ). Should that also be automatically translated in the filesystem?

In German, capitalisation is not reversible. ß becomes SS when capitalised but will be lowercased as ss. (There is now a capital version -- ẞ -- but from what I gather it's not widely used.)

Even in English you have British and American spellings of a given word -- native speakers would recognise them as the same thing but it would not be reasonable to expect a filesystem to map them to the same thing. Initialisms can have different identical representations (N.S.A vs NSA). And you also have cases where capitalisation actually does distinguish words (May vs may, PRISM vs prism, CAT vs cat, etc). What about fullwidth and halfwidth latin characters (Hello vs Hello)? Arguably those are even more identical than upper and lower case.

For all of the above reasons, case insensitivity is something which most systems will only ever implement for English and a few other European languages, meaning that it's more of a wart than a fully-working feature. If the argument really is "well, a human would recognise these two names as the same thing, so the filesystem should too" then why are none of the other examples given above handled? If it's too difficult to do correctly (which is my view) then why support any of this in the first place? However, everything should be normalised (NFC or NFD depending on your usecase).

Re: Unicode Normalization Forms: When ö ≠ ö

#114
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us we would be better off being able to a assert that a piece of text was German, or sanskrit, not a jumble of both. Presumably the person who wrote it speaks a single language. Just because something is not useful to them, it doesn't mean it is not useful…

> Presumably the person who wrote it speaks a single language.

Presumably the person who wrote it speaks English.

Re: Unicode Normalization Forms: When ö ≠ ö

#115

Earlier quoted context omitted.

The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…

> The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Only if different case-variants do not have meaning. When two words that differ only in case have different meaning, we distinguish them (e.g. "moon" and "Moon").

Assuming you mean the distinction between moons in general and the Earth's Moon (Luna), if I wrote "neil armstrong was the first human to walk on the moon" would you think I meant anything other than Neil Armstrong walking on The Moon?

Meaning doesn't go when the case changes in anything like the way meaning goes when the letters change. "neil armstrong was the first human to walk on the roof" is a very different sentence, you can't get anything like that difference with just case changes. If I spoke it, you wouldn't be able to tell if I spoke the correct case or not. Would you want school children searching for "one small step for man, one giant leap for mankind" and Google says "no results" because they used a lowercase m in mankind? Would you want a TV quiz show asking "What is Europa?" and a contenstant answers "a moon of jupiter" and the host asks "do you mean moon with a capital m or lowercase m?" before they decide whether the answer is correct?

Re: Unicode Normalization Forms: When ö ≠ ö

#116
post #50
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

Heh, funny, I'm implementing this exact thing at the moment, oddly enough -- rather, implementing a security check that provides that same guarantee you mention, Mixed Script protections. In Unicode spec terms, 'UTS 39 (Security)' contains the description of how to do this, mostly in section 5, and it relies on 'UTX 24 (Scripts)'. It's more nuanced than your example but only slightly. If you replace "German" with "Ja…

[deleted]

Re: Unicode Normalization Forms: When ö ≠ ö

#117
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> It might have been better if the 'code pages' idea was refined instead of eliminated

Obviously yes, it would have been better.

But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet. So here we are.

Re: Unicode Normalization Forms: When ö ≠ ö

#118
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> but for most of us we would be better off That's simple - it is provably wrong. While relatively uncommon there are plenty of examples that would contradict this statement. And it's not about being able to encode the Rosetta Stone - non-scientists mix languages all the time, from Carmina Burana to Blinkenlights. They even make meaningful portmanteau words and write them with characters from multiple unrelated writi…

You miss the point. The basic unit of ASCII v2 (aka 'Unicode') should have been the codepage, not the codepoint. Having a stateful stream of codepage-symbol pairs is not a problem - in practice, all Unicode encodings ended up being stateful anyways, except in a shitty way that doesn't help to encode any semantic information.

Re: Unicode Normalization Forms: When ö ≠ ö

#119
post #44
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

You can already map Unicode ranges to "code pages" of sorts, so how would that help? Thing is, people who are not linguists do want to mix languages. It's very common in some cultures to intersperse the native language with English. But even if not, if the language in question uses a non-Latin alphabet, there are often bits and pieces of data that have to be written down in Latin. So that "most of us" perspective is…

I am trying to formalise this with Cosmopolitan Identifiers (https://obua.com/publications/cosmo-id/3/). These identifiers consist of words and symbols. Symbols are normalised based on how they look like, and so Latin / Cyrillic / Greek symbols that look alike are mapped to the same symbol. Words are normalised differently, so that "Tree" and "tree" map to the same normal form. As a symbol, "T" and "t" are obviously different. I am not totally happy with the concept yet, I have implemented a fourth, simpler iteration of that concept as a Typescript package: https://www.npmjs.com/package/cosmo-id .

One of the problems is, how do you distinguish symbols and words? A simple way to do this is to classify something as a symbol if it is just a single character, and as a word otherwise. For example, "α-β" would consist of two symbols, separated by a hyphen, but "αβ" is a word and normalised to "av" based on some convention on how to "latinise" greek words.

Re: Unicode Normalization Forms: When ö ≠ ö

#120
post #40

Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…

> It might have been better if the 'code pages' idea was refined instead of eliminated Obviously yes, it would have been better. But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet. So here we are.

> But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet.

This is not even remotely true.

Post reply on HN