Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
Reminds me of the good old days with EUC-KR, KSC 5601, and all those different encoding schemes I've successfully repressed in my memory for years. Yes, you could probably assert that a piece of string was either Korean or English but never anything else... because the system was incapable of representing it. I'm not exactly sure how a code page is supposed to help us here. Developers have trouble supporting multiple…
Unicode Normalization Forms: When ö ≠ ö
111–120 of 144 posts
Re: Unicode Normalization Forms: When ö ≠ ö
#112Earlier quoted context omitted.
You can already map Unicode ranges to "code pages" of sorts, so how would that help? Thing is, people who are not linguists do want to mix languages. It's very common in some cultures to intersperse the native language with English. But even if not, if the language in question uses a non-Latin alphabet, there are often bits and pieces of data that have to be written down in Latin. So that "most of us" perspective is…
Yeah, Greek alphabet is used a lot in sciences. It's really annoying that we're only starting to get proper support now . (Including on keyboards : http://norme-azerty.fr/en/ )
Re: Unicode Normalization Forms: When ö ≠ ö
#113Earlier quoted context omitted.
Case insensitivity is a braindead behavior. If desired it should be a fallback path selecting the best match, not the first resort.
The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…
In Japanese, くるま, クルマ, and 車 are all the same word (the first two are the phonetic spelling "kuruma", the later is the Chinese character). However in order to know that 車 is read くるま you need to be a native Japanese speaker (or have a dictionary) -- should filesystems have dictionaries to match what a human would think? Search engines that support Japanese have to handle this to some degree, but I humbly suggest implement Google Search's language handling code into a filesystem would be an ill-advised decision.
If you wanted to implement the most minimal version of this you would map between katakana and hiragana, but that means you'll need to do this for other languages. For instance, Serbian. Serbian uses two scripts (both of which have upper and lower case forms) and any native Serbian speaker would see "tuđa ljuta paprika" and "туђа љута паприка" as the same text (note that lj became љ). Should that also be automatically translated in the filesystem?
In German, capitalisation is not reversible. ß becomes SS when capitalised but will be lowercased as ss. (There is now a capital version -- ẞ -- but from what I gather it's not widely used.)
Even in English you have British and American spellings of a given word -- native speakers would recognise them as the same thing but it would not be reasonable to expect a filesystem to map them to the same thing. Initialisms can have different identical representations (N.S.A vs NSA). And you also have cases where capitalisation actually does distinguish words (May vs may, PRISM vs prism, CAT vs cat, etc). What about fullwidth and halfwidth latin characters (Hello vs Hello)? Arguably those are even more identical than upper and lower case.
For all of the above reasons, case insensitivity is something which most systems will only ever implement for English and a few other European languages, meaning that it's more of a wart than a fully-working feature. If the argument really is "well, a human would recognise these two names as the same thing, so the filesystem should too" then why are none of the other examples given above handled? If it's too difficult to do correctly (which is my view) then why support any of this in the first place? However, everything should be normalised (NFC or NFD depending on your usecase).
Re: Unicode Normalization Forms: When ö ≠ ö
#114Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
> Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us we would be better off being able to a assert that a piece of text was German, or sanskrit, not a jumble of both. Presumably the person who wrote it speaks a single language. Just because something is not useful to them, it doesn't mean it is not useful…
Presumably the person who wrote it speaks English.
Re: Unicode Normalization Forms: When ö ≠ ö
#115Earlier quoted context omitted.
The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Computers not aligning with what humans do is annoying and frustrating; they should be tools for us, not us for them. There's no way two people would write ö ö and have readers think they were different because one was written in oil-based ink and one…
> The opposite; case insensitivity is what human brains do, we read word WORD Word and woRD as the same thing, it's computer case-sensitive matching which is "brainless". Only if different case-variants do not have meaning. When two words that differ only in case have different meaning, we distinguish them (e.g. "moon" and "Moon").
Meaning doesn't go when the case changes in anything like the way meaning goes when the letters change. "neil armstrong was the first human to walk on the roof" is a very different sentence, you can't get anything like that difference with just case changes. If I spoke it, you wouldn't be able to tell if I spoke the correct case or not. Would you want school children searching for "one small step for man, one giant leap for mankind" and Google says "no results" because they used a lowercase m in mankind? Would you want a TV quiz show asking "What is Europa?" and a contenstant answers "a moon of jupiter" and the host asks "do you mean moon with a capital m or lowercase m?" before they decide whether the answer is correct?
Re: Unicode Normalization Forms: When ö ≠ ö
#116Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
Heh, funny, I'm implementing this exact thing at the moment, oddly enough -- rather, implementing a security check that provides that same guarantee you mention, Mixed Script protections. In Unicode spec terms, 'UTS 39 (Security)' contains the description of how to do this, mostly in section 5, and it relies on 'UTX 24 (Scripts)'. It's more nuanced than your example but only slightly. If you replace "German" with "Ja…
Re: Unicode Normalization Forms: When ö ≠ ö
#117Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
Obviously yes, it would have been better.
But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet. So here we are.
Re: Unicode Normalization Forms: When ö ≠ ö
#118Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
> but for most of us we would be better off That's simple - it is provably wrong. While relatively uncommon there are plenty of examples that would contradict this statement. And it's not about being able to encode the Rosetta Stone - non-scientists mix languages all the time, from Carmina Burana to Blinkenlights. They even make meaningful portmanteau words and write them with characters from multiple unrelated writi…
Re: Unicode Normalization Forms: When ö ≠ ö
#119Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
You can already map Unicode ranges to "code pages" of sorts, so how would that help? Thing is, people who are not linguists do want to mix languages. It's very common in some cultures to intersperse the native language with English. But even if not, if the language in question uses a non-Latin alphabet, there are often bits and pieces of data that have to be written down in Latin. So that "most of us" perspective is…
One of the problems is, how do you distinguish symbols and words? A simple way to do this is to classify something as a symbol if it is just a single character, and as a word otherwise. For example, "α-β" would consist of two symbols, separated by a hyphen, but "αβ" is a word and normalised to "av" based on some convention on how to "latinise" greek words.
Re: Unicode Normalization Forms: When ö ≠ ö
#120Reading about unicode has made me much, much more circumspect about the meaning of != in languages, and what fall-through behavior should look like. Unicode domain names lasted for a hot minute until someone registered microsoft.com with Cyrillic letters. Years ago I read a rant by someone who insisted that being able to mix arbitrary languages into a single String object makes sense for linguists but for most of us…
> It might have been better if the 'code pages' idea was refined instead of eliminated Obviously yes, it would have been better. But Unicode was designed by the same people who designed ASCII - monolingual Americans who never had to deal on a daily basis with anything that doesn't fit into the 26 letters of the English alphabet. So here we are.
This is not even remotely true.