The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
devblogs.microsoft.com
The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
1–10 of 73 posts
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#2I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps.
It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They could probably do it in time for Windows XP!
Instead, UTF-8 codepage was only introduced in Windows 7 (but had incomplete support), and finally got the proper support in Windows 10. That's a lot of time Windows programmers were forced to use UCS-2.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#3I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps. It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They…
Maybe they didn't want to be early adopters of what could turn out to be a second fad like UCS-2 was the first time.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#4The fact that in Windows-speak 'unicode' and 'UCS-2' (an encoding) have been used as synonyms has caused a universe of confusion. Even after I grokked the difference I still had arguments with colleagues who remained bamboozled by it. The fact that for a brief time there were only 64k code points, so that you could sort-of fudge the distinction, only made things worse in the long run.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#5The fact that in Windows-speak 'unicode' and 'UCS-2' (an encoding) have been used as synonyms has caused a universe of confusion. Even after I grokked the difference I still had arguments with colleagues who remained bamboozled by it. The fact that for a brief time there were only 64k code points, so that you could sort-of fudge the distinction, only made things worse in the long run.
In my experience, most technical people these days still don't know the difference between UTF-8 and Unicode for example.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#6The fact that in Windows-speak 'unicode' and 'UCS-2' (an encoding) have been used as synonyms has caused a universe of confusion. Even after I grokked the difference I still had arguments with colleagues who remained bamboozled by it. The fact that for a brief time there were only 64k code points, so that you could sort-of fudge the distinction, only made things worse in the long run.
In my experience, most technical people these days still don't know the difference between UTF-8 and Unicode for example.
Is this the best reference? Too detailed?
https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#7(2019)
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#8I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps. It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They…
Maybe they didn't want to be early adopters of what could turn out to be a second fad like UCS-2 was the first time.
What's depressing about the saga of utf8 adoption is that it's such an obvious solution. The ISO2022 standards used the principle since the 70s, and programmers of the era were widely accustomed to extending 8 bit instruction sets to multibyte instructions.
It's one of those cases where you need to hit your head hard and repeatedly until you bite the bullet and accept that the cost of multibyte (for example, strlen() different from character count, need to parse to seek etc.) are trivial compared to the benefits.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#9I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps. It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They…
I have asked myself the same question. However, I observed that the switch from UCS-2 to true UTF-16 was already very bumpy, with strange bugs and tons of edge cases that only got significantly better in Windows 8. A forced switch to UTF-8 would likely have broken ANSI strings.
Languages such as PHP didn't get true UTF-8 support until 2015, Python in 2008 (and that switch took a decade), and Java and JavaScript continue to use UTF-16 as default strings.
The wastefulness of UTF-16 is mostly remedied by Unicode Compression (SCSU), which can be much shorter than UTF-8 in languages like Hindi and Chinese.
Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)
#10I remember this hoopla.
And windows wasn't the only victim of using "two bytes for a char" for unicode support.
Java fell in this trap too. The team was split in deciding "whether to waste a single byte per character for string" -- that's in their words, and then decided to rather be ready for the future. That's in 1995.
And then everyone later watched UTF-8 take over the world.
The PHP team also wasted their version 6 and a few years of development effort trying to support multi byte chars by ditching all previous code. And then failed. There was never a version 6. They continued with the old code base with version 7.
Why didn't they go the UTF-8 round? May be Unicode consortium was late to introduce it. Maybe something else.