The fact that in Windows-speak 'unicode' and 'UCS-2' (an encoding) have been used as synonyms has caused a universe of confusion. Even after I grokked the difference I still had arguments with colleagues who remained bamboozled by it. The fact that for a brief time there were only 64k code points, so that you could sort-of fudge the distinction, only made things worse in the long run.
Note that Windows uses UTF-16, not UCS-2.
The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
41–50 of 73 posts
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#42Earlier quoted context omitted.
In my experience, most technical people these days still don't know the difference between UTF-8 and Unicode for example.
Is this the best reference? Too detailed? https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#43Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#44Earlier quoted context omitted.
What's depressing about the saga of utf8 adoption is that it's such an obvious solution. The ISO2022 standards used the principle since the 70s, and programmers of the era were widely accustomed to extending 8 bit instruction sets to multibyte instructions. It's one of those cases where you need to hit your head hard and repeatedly until you bite the bullet and accept that the cost of multibyte (for example, strlen()…
UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. This is why UTF8 wasn't universally hailed to begin with. Countries like Japan had already solved encoding differently while US was still using ASCII. (See JIS encoding). As time passed tho, the byte savings matter less. Also combined with transfer compression makes the encoding size differences n…
In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all in single-byte characters.
Of course, there are still many scenarios where UTF-16 "wins" in terms of size – especially with plain text – but real-world size comparisons tend to be a bit more tricky.
For example, for the current Wikipedia homepage of ja, zh, and ko:
% file *
ja-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2709)
ja-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2709)
ko-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2503)
ko-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2656)
zh-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (5589)
zh-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (5589)
% ls -lh
210K ja-16.htm
118K ja-8.htm
193K ko-16.htm
106K ko-8.htm
189K zh-16.htm
104K zh-8.htm
The UTF-16 versions are all larger (and that's just the HTML, excluding the CSS, JS, etc).I wrote a small script to count the number of "wide" multibyte characters:
ja-8.htm 100,808 7-bit characters; 19,026 wide characters
ko-8.htm 93,888 7-bit characters; 14,149 wide characters
zh-8.htm 91,589 7-bit characters; 14,360 wide characters
It was higher than I expected, although not that surprising when looking at the
source when you have things like:
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#45Earlier quoted context omitted.
Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.
UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs. For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#46I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps. It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They…
IIRC, the problem was that a MBCS codepage could have a maximum of 2 bytes per character, while UTF-8 could need 3 or even 4 bytes per character. Legacy applications could have fixed-size buffers with space for only 2 bytes per character, and would break if the default codepage needed more than that.
> Instead, UTF-8 codepage was only introduced in Windows 7 (but had incomplete support), and finally got the proper support in Windows 10. That's a lot of time Windows programmers were forced to use UCS-2.
It appears to me that lately Windows hasn't been as obsessed with backwards compatibility as it had been in the past. They are also gradually allowing for more use of paths longer than MAX_PATH (260 characters), which are not accessible to legacy applications, and as everyone knows, the compatibility with 16-bit applications has been dropped some time ago.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#47Earlier quoted context omitted.
I have asked myself the same question. However, I observed that the switch from UCS-2 to true UTF-16 was already very bumpy, with strange bugs and tons of edge cases that only got significantly better in Windows 8. A forced switch to UTF-8 would likely have broken ANSI strings. Languages such as PHP didn't get true UTF-8 support until 2015, Python in 2008 (and that switch took a decade), and Java and JavaScript conti…
How many Windows users do benefit significantly from using UCS-2 encoding, I wonder? Chinese and Hindi (along with Japanese, South Korean and various Arabic language) speakers would surely come close to outnumbering those of us using European alphabets (I gather that Cyrillic & Greek have single-byte encodings, but maybe they mostly use UCS-2 these days too?). On that basis it seems a reasonable choice as the default…
You are probably right. Just slightly off topic: Hindi speakers who use computers mostly seem happy enough with English-language forms, instructions, and technical documents. This is true of most Indian users who speak languages that use one of the Indic scripts. Actual non-English text is useful mainly for publications like newspapers, magazines, and books, which native speakers do prefer to consume in their own language.
The above is, of course, not counting the significant numbers of Indians who are quite comfortable using English for daily communication. These Anglophones are just like US speakers in their preferences and have no use for internationalization.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#48The fact that in Windows-speak 'unicode' and 'UCS-2' (an encoding) have been used as synonyms has caused a universe of confusion. Even after I grokked the difference I still had arguments with colleagues who remained bamboozled by it. The fact that for a brief time there were only 64k code points, so that you could sort-of fudge the distinction, only made things worse in the long run.
In my experience, most technical people these days still don't know the difference between UTF-8 and Unicode for example.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#49I remember this hoopla. And windows wasn't the only victim of using "two bytes for a char" for unicode support. Java fell in this trap too. The team was split in deciding "whether to waste a single byte per character for string" -- that's in their words, and then decided to rather be ready for the future. That's in 1995. And then everyone later watched UTF-8 take over the world. The PHP team also wasted their version…
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#50Earlier quoted context omitted.
Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.
This is plain decoding of a trusted string: utf16 decode if (unit = 0xE000) /* one unit */; else /* two units */ utf-8 decode if (unit UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding: utf16 encode if (value (We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.) Things get even more complex when we are to read an untrusted string…
utf-8 decode
switch (std::countl_one(unit)) {
case 0:
/* one unit */
break;
case 2:
/* two units */
break;
case 3:
/* three units */
break;
case 4:
/* four units */
break;
default:
/* not code point boundary */
break;
}