Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Every Software Developer Must Know About Unicode (2003)

joelonsoftware.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: What Every Software Developer Must Know About Unicode (2003)

#31

Post by mau »

AnthonyMouse wrote: post #28
gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

> The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article). Why has Microsoft still not fixed this? They have the various functions [Abc] which call [Abc]A and [Abc]W depending on whether UNICODE is defined, it seems obvious that they should add [Abc]U for UTF-8 and provide a similar de…

>> The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article).

> Why has Microsoft still not fixed this?

Is there a way they can fix it without breaking backward compatibility?

Re: What Every Software Developer Must Know About Unicode (2003)

#32

Post by jwr »

gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

I started treating what Joel says much less seriously after discovering that FogBugz sends E-mail using "windows-1252" or "windows-1250" character sets instead of UTF-8. I reported this as a problem to customer service, but was told that it will not be changed. Quoting from the reply I got "joelonsoftware.com is a blog. Fog Creek Software is a business."

Some people might give Joel additional credibility because of his successful business — but there seems to be a gap between theory and practice, which undermines that credibility, at least for me.

Re: What Every Software Developer Must Know About Unicode (2003)

#33

Post by lmm »

millstone wrote: post #21
oofabz wrote: post #19

Earlier quoted context omitted.

I know that both encodings are variable-length. That is the issue I am trying to address. My point is that in UTF-16 it's too easy to ignore surrogate pairs. Lots of UTF-16 software fails to handle variable-length characters because they are so rare. But in UTF-8 you can't ignore multi-byte characters without obvious bugs. These bugs are noticed and fixed more quickly than UTF-16 surrogate pair bugs. This makes UTF-8…

Bugs in UTF-8 handling of multibyte sequences need not be obvious. Google "CAPEC-80." UTF-16 has an advantage in that there's fewer failure modes, and fewer ways for a string to be invalid. edit: As for surrogate pairs, this is an issue, but I think it's overstated. A naïve program may accidentally split a UTF-16 surrogate pair, but that same program is just as liable to accidentally split a decomposed character sequ…

> A naïve program may accidentally split a UTF-16 surrogate pair, but that same program is just as liable to accidentally split a decomposed character sequence in UTF-8. You have to deal with those issues regardless of encoding.

The point is that using UTF-8 makes these issues more obvious. Most programmers these days think to test with non-ascii characters. Fewer think to test with astral characters.

Re: What Every Software Developer Must Know About Unicode (2003)

#34

Post by mhp »

jwr wrote: post #32
gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

I started treating what Joel says much less seriously after discovering that FogBugz sends E-mail using "windows-1252" or "windows-1250" character sets instead of UTF-8. I reported this as a problem to customer service, but was told that it will not be changed. Quoting from the reply I got "joelonsoftware.com is a blog. Fog Creek Software is a business." Some people might give Joel additional credibility because of h…

I'm sorry you got that reply. I don't doubt you received it, but it doesn't reflect how I believe my company should have responded to you.

This is actually a feature. FogBugz DOES send email in UTF-8 (just copy some hebrew text and send an email to yourself) if it has to. If it doesn't have to, it will not do so simply because it's working around old email clients that don't support UTF-8. FogBugz has supported all kinds of languages for many, many years, so there's some possibility you are talking about a reply you received before that even happened, but I think you just observed a normal English only email from FogBugz has the content-type set to windows-1252. You could argue at this point that we probably don't need that workaround anymore I guess. I just have to weigh that against whether it's broken now (it's not) vs. the new support emails we would get from people who were having problems with email because they were still using Eudora 3.0.

It turns out that it's actually BECAUSE of the whole reason Joel wrote that article that FogBugz sends English email in windows-1252.

Re: What Every Software Developer Must Know About Unicode (2003)

#35

Post by ygra »

mau wrote: post #31
AnthonyMouse wrote: post #28

Earlier quoted context omitted.

> The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article). Why has Microsoft still not fixed this? They have the various functions [Abc] which call [Abc]A and [Abc]W depending on whether UNICODE is defined, it seems obvious that they should add [Abc]U for UTF-8 and provide a similar de…

>> The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article). > Why has Microsoft still not fixed this? Is there a way they can fix it without breaking backward compatibility?

They can, by introducing a few thousand stub functions which convert and delegate to ★W functions, but there'd be little use to do so. Every sane program out there uses the ★W functions and some insane still use the ★A ones. So it would only be beneficial for new code while all existing code remains the same, with the same encoding bugs if there are any.

I'm also not sure whether there are that many cases where it really helps. UTF-8 only on Windows is painful and so is using UTF-8 only with all other things that use UTF-16 (Qt, Java, etc.). Usually in those cases you use a library/framework/whatever that handles the platform abstraction or just conform to what's expected.

Re: What Every Software Developer Must Know About Unicode (2003)

#36

Post by lucian1900 »

userbinator wrote: post #23
lucian1900 wrote: post #17

Earlier quoted context omitted.

There is absolutely no situation in which UTF-16 wins over UTF-8, because of the surrogate pairs required. That makes both encodings variable length. UTF-32 is probably what you're thinking of.

UTF-8 is variable length in that it can be anywhere from 1 to 4 bytes, while UTF-16 can either be 2 or 4. That makes a UTF-16 decoder/encoder half as complex as a UTF-8 one.

Surrogate pairs are way more complex than anything in UTF-8.

Re: What Every Software Developer Must Know About Unicode (2003)

#37

Post by AnthonyMouse »

ygra wrote: post #35
mau wrote: post #31

Earlier quoted context omitted.

>> The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article). > Why has Microsoft still not fixed this? Is there a way they can fix it without breaking backward compatibility?

They can, by introducing a few thousand stub functions which convert and delegate to ★W functions, but there'd be little use to do so. Every sane program out there uses the ★W functions and some insane still use the ★A ones. So it would only be beneficial for new code while all existing code remains the same, with the same encoding bugs if there are any. I'm also not sure whether there are that many cases where it re…

> Every sane program out there uses the ★W functions and some insane still use the ★A ones. So it would only be beneficial for new code while all existing code remains the same, with the same encoding bugs if there are any.

Every sane program uses the undifferentiated functions, defines UNICODE so that they map to the ★W functions, and uses TCHAR which defining UNICODE causes to map to a wide char. Older programs don't define UNICODE and often use char (or CHAR) instead of TCHAR. If they would create a different define (e.g. '#define UTF8') which would map to the new UTF8 functions and would define TCHAR as CHAR then anything doing it either way would do the right thing just by defining UTF8 and recompiling. Only programs that explicitly call the ★W functions (which they never should have exposed) wouldn't be "fixed" to use UTF8, but neither would they be broken.

> UTF-8 only on Windows is painful

...because Microsoft hasn't fixed it.

> Usually in those cases you use a library/framework/whatever that handles the platform abstraction or just conform to what's expected.

That's a cop out. You're just deferring to the frameworks, which also shouldn't be using anything other than UTF8, and who may have more difficulty in fixing it because the transition mechanism Microsoft used to unicode is well adaptable to another transition. Not every library you have to use will use the same encoding as the framework and you're back to a huge pain. The only way to fix it is for everything to always use UTF8, and deprecate everything else going forward.

Re: What Every Software Developer Must Know About Unicode (2003)

#38

Post by lambda »

kmontrose wrote: post #29
lambda wrote: post #27

Earlier quoted context omitted.

The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 Sadly, that's not really correct. Core Foundation and Cocoa on Mac OS X use UTF-16. Qt uses UTF-16. Java uses UTF-16. JavaScript uses UTF-16. Many of these APIs have easy methods for converting to and from other encodings, and offer a certain amount of abstraction over the underlying encoding, but it still shows th…

There's a reason a ton of legacy code is UCS-2^/UTF-16 and not UTF-8. The code is older than UTF-8. Windows NT was first released in '93, UTF-8 didn't exist (much less have wide adoption) for most of it's development. Likewise, NeXTSTEP had an initial release in '89. Java and JavaScript (both '95) could have adopted UTF-8, but they're almost certainly running on an OS that expects UTF-16 soooo... yeah. ^UCS-2 was sup…

It doesn't matter when the projects started, it matters when they added Unicode support. Windows NT didn't support Unicode until 4.0, released in 1996. NeXTSTEP may have been released in '89, but Unicode itself wasn't finished until '91, it couldn't possibly have supported Unicode upon release. The original releases of NeXTSTEP just used C strings; it wasn't until OpenStep in 1994 that they introduced NSString based on UCS-2.

UTF-8 was publicly released in January 1993. So by the time these projects became Unicode enabled, UTF-8 had already existed for at least a year.

Java and JavaScript had no underlying platform constraints to choose UCS-2/UTF-16, since the underlying platforms didn't support Unicode during their development.

Qt 2.0 was the first release of Qt to introduce Unicode support in QString, and it was released in 1999.

No, the real problem was just the fundamental design mistake that the Unicode consortium made when first developing Unicode. They thought that 16 bits would be enough to fit all of the world's actively used writing systems, and the simplest way to support an extended character set would be to just switch the underlying character type from 8 bit integers to 16 bit integers. This was a mistake in many ways; 16 bits is not sufficient, especially when CJK is taken into account, and so they had to do a lot of unification that wasn't really appropriate and led to a lot of resistance to using Unicode from CJK users. Changing to 16 bit integers for the fundamental character type meant that every API had to be duplicated to provide a wide character version. Some APIs already had wide character support for legacy wide character sets, but differences in existing wide character support between NT (which used 16 bit wide characters) and many Unices (which used 32 bit wide characters) meant that writing portable code is quite difficult. Using 16 bit integers for an internal representation means that there's a native endianness, but once you need to interchange data endianness becomes a big issue. And so on.

UTF-8 was the solution to many of these problems, and it was introduced before Unicode support had become widespread, but the idea that 16 bit types should be used for Unicode had already permeated people's consciousness and likely early development efforts. It's too bad that more people didn't learn from Plan 9's experience switching to UTF-8, which happened all the way back in 1992 (they switched Plan 9 to UTF-8 before publicly announcing it, which acted as a very good proof of concept).

Re: What Every Software Developer Must Know About Unicode (2003)

#39

Post by Someone »

lucian1900 wrote: post #17
oofabz wrote: post #11

Earlier quoted context omitted.

I think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties. UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, beca…

There is absolutely no situation in which UTF-16 wins over UTF-8, because of the surrogate pairs required. That makes both encodings variable length. UTF-32 is probably what you're thinking of.

Anything in the range U+0800 to U+FFFF takes three bytes per character in UTF-8 and two in UTF-16 (http://en.wikipedia.org/wiki/Comparison_of_Unicode_encodings...:

"Therefore if there are more characters in the range U+0000 to U+007F than there are in the range U+0800 to U+FFFF then UTF-8 is more efficient, while if there are fewer then UTF-16 is more efficient. "

That same page also states: "A surprising result is that real-world documents written in languages that use characters only in the high range are still often shorter in UTF-8, due to the extensive use of spaces, digits, newlines, html markup, and embedded English words", but I think the "citation needed]" is added rightfully there (it may be close in many texts, though)

Post reply on HN