Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

devblogs.microsoft.com

Post reply on HN
Print view View original

21–30 of 73 posts

Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

#21

Post by gpderetta »

Cloudef wrote: post #19
habibur wrote: post #10

I remember this hoopla. And windows wasn't the only victim of using "two bytes for a char" for unicode support. Java fell in this trap too. The team was split in deciding "whether to waste a single byte per character for string" -- that's in their words, and then decided to rather be ready for the future. That's in 1995. And then everyone later watched UTF-8 take over the world. The PHP team also wasted their version…

UTF-8 was plan9's invention, I don't think anybody saw it coming.

For more details: Rob Pike & Ken Thompson invented UTF-8 when prompted by X/Open (Not sure if X/Open and POSIX were already merged at that point) and they first implemented the proof of concept on plan9 (https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt).

Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

#22

Post by heinrich5991 »

Mikhail_Edoshin wrote: post #20
ainar-g wrote: post #17

Earlier quoted context omitted.

I feel like you're confusing it with UTF-32, which is fixed-length. UTF-16 is neither fixed-length, like UTF-32, nor compatible with the old ASCII encoding, like UTF-8. The historical reasons of why UTF-16 exists are fully understandable, but by now it is clear that it's the worst of both worlds.

No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibili…

Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.

Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

#23

Post by nikbackm »

fredoralive wrote: post #16
theamk wrote: post #2

I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps. It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They…

IIRC the attitude was that the "ASCII" versions of functions largely existed only for backwards compatibility, after all on NT[1] they're basically just wrappers that do character set conversion before passing through to the real Unicode (UCS2/UTF16) functions. The view was that if you want Unicode, you should use the real functions directly, which means 16 bit strings. I think there was also some compatibility conce…

The "A" stands for "ANSI", not "ASCII". Otherwise largely correct.

Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

#26

Post by bigDinosaur »

Mikhail_Edoshin wrote: post #20
ainar-g wrote: post #17

Earlier quoted context omitted.

I feel like you're confusing it with UTF-32, which is fixed-length. UTF-16 is neither fixed-length, like UTF-32, nor compatible with the old ASCII encoding, like UTF-8. The historical reasons of why UTF-16 exists are fully understandable, but by now it is clear that it's the worst of both worlds.

No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibili…

Can you show some concrete instances of it being easier to work with compared to UTF-8? As far as I can see you gain variable length encoding (albeit in a potentially confusing way) while losing ASCII support and also not gaining the pure simplicity of UTF-32. Oh and also endianness, although personally that's probably never an issue I will run into.

Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

#27

Post by Const-me »

heinrich5991 wrote: post #22
Mikhail_Edoshin wrote: post #20

Earlier quoted context omitted.

No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibili…

Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.

UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs.

For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.

Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

#28

Post by Grame1983 »

bigDinosaur wrote: post #26
Mikhail_Edoshin wrote: post #20

Earlier quoted context omitted.

No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibili…

Can you show some concrete instances of it being easier to work with compared to UTF-8? As far as I can see you gain variable length encoding (albeit in a potentially confusing way) while losing ASCII support and also not gaining the pure simplicity of UTF-32. Oh and also endianness, although personally that's probably never an issue I will run into.

[dead]

Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

#29

Post by ChrisSD »

jbverschoor wrote: post #24

> Windows adopted Unicode before most other operating systems. [citation needed] Windows multibyte / mbcs is not unicode.

Windows NT uses actual Unicode under the hood (originally UCS-2, now UTF-16).

Windows NT started development in 1989 and was released in 1993. Considering the Unicode standard was first published in 1991/2, I think Windows NT can be counted as an early adopter.

Re: The sad history of Unicode printf-style format specifiers in Visual C++ (2019)

#30

Post by Mikhail_Edoshin »

heinrich5991 wrote: post #22
Mikhail_Edoshin wrote: post #20

Earlier quoted context omitted.

No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibili…

Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.

This is plain decoding of a trusted string:

    utf16 decode
      if (unit = 0xE000)
        /* one unit */;
      else
        /* two units */
    utf-8 decode
      if (unit
UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding:

    utf16 encode
      if (value
(We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.)

Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats:

    UTF-8    : 13 byte types, 13 states, 10 actions
    UTF-16LE :  3 byte types,  4 states,  5 actions
Post reply on HN