61–70 of 73 posts
I remember this hoopla. And windows wasn't the only victim of using "two bytes for a char" for unicode support. Java fell in this trap too. The team was split in deciding "whether to waste a single byte per character for string" -- that's in their words, and then decided to rather be ready for the future. That's in 1995. And then everyone later watched UTF-8 take over the world. The PHP team also wasted their version…
Earlier quoted context omitted.
> Why didn't they go the UTF-8 round? The original Unicode marketing / position statement from 1988[1] may provide a clue: “In the Unicode system, a simple unambiguous fixed-length character encoding is integrated into a coherent overall architecture for text processing.” “Unicodes [sic] are the most straightforward multilingual generalization of ASCII codes: - Fixed length of character code (16 bits); [...]” “Are 16…
Do you (or anyone) have some idea why anyone could possibly have thought 16 bits would be enough? Many decisions are bad in hindsight but surely no hindsight was needed for that.
If you look at the text following the “Yes” quote, you’ll find that “all characters” is carefully defined to mean ”all characters in current use from commercially non-negligible scripts”. Compared to the current definition of “all characters we have reasonable evidence have ever been used for natural-language interchange”, it doesn’t sound as noble, but would also exclude a number of large-repertoire sets (Tangut and pre-modern Han ideograms, Yi syllables, hieroglyphs, cuneiform). Remove the requirement for 1:1 code point mapping with legacy sets, and you could conceivably throw out precomposed Hangul as well. (Precomposed European scripts too, if you want, but that wouldn’t net you eleven thousand codepoints.)
At that point the question seems to come down to Han characters: the union of all government-mandated education standards (unified) would come down well below ten thousand characters, but how well does that number correspond to the number of characters people actually need? One potential source of death is uncommon characters people really, really want (proper names), but overall, I don’t know, you’d probably need a CJKV expert to tell. To me, neither answer seems completely implausible.
On the other hand, it’s also unclear that a constant-width encoding would really be all that valuable. Most of the time, you are either traversing all code points in sequence or working with larger units such as combining-character sequences or graphemes, so aside from buffer truncation issues constant width does not really help all that much. But that’s an observation that took more than a decade of Unicode implementations to crystallize.
It is certainly annoying how large and sparse the lookup tables needed to implement a current version of Unicode are—enough that you need three levels in your radix tree and not two—but if you aren’t doing locales it’s still a question of at most several dozens of kilobytes, not really a deal breaker these days. Perhaps that’s not too much of a cost for not marginalizing users of obscure languages and keeping digitized historical text representable in the common format.
I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps. It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They…
> It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They could probably do it in time for Windows XP! IIRC, the problem was that a MBCS codepage could have a maximum of 2 bytes per character, while UTF-8 could need 3 or even 4 bytes per charac…
Make "SetThreadLocale" or something work with UTF-8, so apps could switch one-by-one (this may not work with hooks but still better than nothing...). Add UTF-8 support to MSVC runtime, so at least fopen can be safe (people have been asking for it forever). Add FILE_FLAG_UTF_8 to CreateFileA. Have Notepad be able to edit UTF-8.
But I suspect the real reason is there was a strong internal opposition to UTF-8 -- maybe it was seen as inferior to UTF-16, maybe it was NIH, maybe they hoped the world will change to follow Windows ways and to put BOM marks in each document...
Oh well, I am glad they have at least changed now.
Earlier quoted context omitted.
Windows NT uses actual Unicode under the hood (originally UCS-2, now UTF-16). Windows NT started development in 1989 and was released in 1993. Considering the Unicode standard was first published in 1991/2, I think Windows NT can be counted as an early adopter.
Well, UCS-2 is not UTF-16
Earlier quoted context omitted.
Your argument uses a website whose code was written by English speakers. There would still be ASCII but verbose element names like "vector-page-toolbar-container" would definitely be shorter both in utf-8 and in utf-16 if they weren't written in English
is still , no matter the language of the author. As is "background-color: #fff" in CSS. The UTF-16 is almost double the size, so you'd have to replace a lot of identifiers with 2 or 3 byte ones. Plus many identifiers come from libraries, and when creating their own identifiers many people use either full English or partial English no matter what language (it was a huge mistake to not use English for many identifiers…
Earlier quoted context omitted.
Just wanted to say thank you for linking these documents! I find the history of character encoding design so interesting, especially from primary sources such as these. :)
Well then :) You might also like the report “Character Set Issues for Ada 9X”[1] from 1989, the only online source I know that goes into any sort of detail about the old ISO 10646 draft before it was essentially replaced[2,3] with Unicode. (Other references about that beast are welcome!) The only trace of it in current use looks to be the term “plane” for a naturally-aligned set of 2^16 code points, originally from a…
Every time I read about the history of character encodings I feel like I learn about a new encoding standard that attempted to standardize things. Reading about this led me to reading more about ASCII as well. I learned it was derived from the 1924 ITA2 standard which was itself derived from the "Baudot" printing telegraph encoding from 1874! It always amazes me how much history surrounds this topic! [1]
Also, that DTIC site is such a treasure trove of great information! :D
Earlier quoted context omitted.
Your argument uses a website whose code was written by English speakers. There would still be ASCII but verbose element names like "vector-page-toolbar-container" would definitely be shorter both in utf-8 and in utf-16 if they weren't written in English
is still , no matter the language of the author. As is "background-color: #fff" in CSS. The UTF-16 is almost double the size, so you'd have to replace a lot of identifiers with 2 or 3 byte ones. Plus many identifiers come from libraries, and when creating their own identifiers many people use either full English or partial English no matter what language (it was a huge mistake to not use English for many identifiers…
You're missing the point entirely, the amount of characters you used is enough for 2 or 3 sentences. This was not an example constructed in good faith.
Earlier quoted context omitted.
This is plain decoding of a trusted string: utf16 decode if (unit = 0xE000) /* one unit */; else /* two units */ utf-8 decode if (unit UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding: utf16 encode if (value (We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.) Things get even more complex when we are to read an untrusted string…
utf-8 has an elegance that utf-16 lacks: utf-8 decode switch (std::countl_one(unit)) { case 0: /* one unit */ break; case 2: /* two units */ break; case 3: /* three units */ break; case 4: /* four units */ break; default: /* not code point boundary */ break; }
Earlier quoted context omitted.
Maybe they didn't want to be early adopters of what could turn out to be a second fad like UCS-2 was the first time.
What's depressing about the saga of utf8 adoption is that it's such an obvious solution. The ISO2022 standards used the principle since the 70s, and programmers of the era were widely accustomed to extending 8 bit instruction sets to multibyte instructions. It's one of those cases where you need to hit your head hard and repeatedly until you bite the bullet and accept that the cost of multibyte (for example, strlen()…