Unicode Normalization Forms: When ö ≠ ö

What about people who CAN read English that way, but think having to match case when searching or referencing text hinders more than it helps?

For languages that have Latin-derived writing systems, it's not uncommon to use English letters (without diacritics) to write the language -- how would that be handled? In addition, thanks to Han unification (though this would've been a problem anyway -- loads of characters would've been unified anyway) all similar CJK Hanzi/Kanji/漢字 characters are mapped to the same codepoint regardless of language. This means that for some sentences it is entirely possible for you to not know whether a sentence fragment is Chinese or Japanese without more context and a native-like understanding of the language.

Also in many languages English words are written verbatim meaning that you can have sentences like (my Japanese is not perfect, this is just an example):

> あの芸人のYouTube動画を見たの？面白すぎるww

And (at least in Japanese) there are loads of other usage of Latin characters aside from loan words that would be too unwieldy to write in katakana -- "w" is like "lol", and fair few acronyms (BGM = Background Music, CM = Advertisement, TKG = 卵かけご飯 = (Raw) Egg on Rice). There are other languages that have similar "issues", but unfortunately I can't give any more examples because the only other language I speak (Serbian) writes everything (even people's names) phonetically.

As an aside -- if anyone ever has to support CJK languages (in subtitles for instance), please make sure you use the right fonts. While Unicode has encoded Han characters with the same codepoint, in different languages the characters are often drawn differently and you need to use the right font for the corresponding language (and area -- Mandarin speakers in different areas write some characters differently -- 返 is different in every CJK locale). Many media players and websites do not handle this correctly and it is fairly frustrating -- the net result is that Japanese is often displayed using Chinese fonts which makes it uncomfortable to read (it's still obvious what the character is, it's just off-putting).

These are all myths, and APIs for such things are bugs. The only thing you can meaningfully do with two pieces of arbitrary Unicode text is to say if they are byte-by-byte equal. For any other operation, you need to have specific business logic.

For example, are "Ionuț" and "Ionut" and "Ionutz" the same string or different strings? There is no generic answer: depending on the intended business logic, they may be identical or not (e.g. if we consider these to be Romanian names, they should be considered identical for search purposes, but probably not identical for storage purposes, where you want to remember exactly how the person spelled their name).

A related problem is that most langauges have no separate types for Text/String on one hand, and Symbol on the other. Text or other Strings should be opaque human text, that can only be interpreted by specific code, offering almost no API (only code point iteration). Symbols should be a restricted subset of Unicode that can offer fuller features, such as lengths, equality, separation into words etc. This would be the type of JSON key names used in serialization and deserialization, for example.