The UTF-8-Everywhere Manifesto

The UTF-8-Everywhere Manifesto
utf8everywhere.org

"UTF-16 is the worst of both worlds—variable length and too wide"

Really, the author tries to convince the reader, but it's not that clean cut.

One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32)

"It's not fixed width" But for most text, it is. Sure, you can do UTF-32 and it may not be a bad idea (today)

Yes, Windows has to deal with several complications and with backwards compatibility, so it's a bag of hurt. Still, they went the right way (internally, it's unicode, period.)

"in plain Windows edit control (until Vista), it takes two backspaces to delete a character which takes 4 bytes in UTF-16"

If I'm not mistaken this is by design. The 4 byte characters is usually typed as a combination of characters, so if you want to change the last part of the combination you jut type one backspace.

I've been through the Windows Unicode stuff at a previous job, which ended up using mostly UTF-16 with some UTF-8 for interfacing to third party libraries and for files which needed to be backward compatible to ASCII (plus significant space savings, which I fought hard for). I think I prefer that approach though, since after the (difficult) conversion you didn't need to worry about encodings in 99% of the code. By their rules you'd gain significant complexity by transforming all over the place in any non-trivial GUI code.

That obviously depends entirely on what kind of application we're talking about. Keeping large amounts of text data in memory as efficiently as possible is one of my greatest concerns. Many people are processing lots of text nowadays, more than ever before.

"It's not fixed width" But for most text, it is.

True, so ignoring it means that your code will be correct ... most of the time.

It is actually not that simple. By using UTF-16 you already have at least two problems:

1. You should know if byte order is big endian or little endian.

2. You should know if your API supports whole unicode set or only 65536 symbols. E.g. Windows API. Do you know answer? What will happen if your user wants to abuse your system by using symbols outside those 65536.