Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Every Software Developer Must Know About Unicode (2003)

joelonsoftware.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: What Every Software Developer Must Know About Unicode (2003)

#21

Post by millstone »

oofabz wrote: post #19
lucian1900 wrote: post #17

Earlier quoted context omitted.

There is absolutely no situation in which UTF-16 wins over UTF-8, because of the surrogate pairs required. That makes both encodings variable length. UTF-32 is probably what you're thinking of.

I know that both encodings are variable-length. That is the issue I am trying to address. My point is that in UTF-16 it's too easy to ignore surrogate pairs. Lots of UTF-16 software fails to handle variable-length characters because they are so rare. But in UTF-8 you can't ignore multi-byte characters without obvious bugs. These bugs are noticed and fixed more quickly than UTF-16 surrogate pair bugs. This makes UTF-8…

Bugs in UTF-8 handling of multibyte sequences need not be obvious. Google "CAPEC-80."

UTF-16 has an advantage in that there's fewer failure modes, and fewer ways for a string to be invalid.

edit: As for surrogate pairs, this is an issue, but I think it's overstated. A naïve program may accidentally split a UTF-16 surrogate pair, but that same program is just as liable to accidentally split a decomposed character sequence in UTF-8. You have to deal with those issues regardless of encoding.

Re: What Every Software Developer Must Know About Unicode (2003)

#22

Post by asifsignupfor »

gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

So when writing Windows applications with Win32, would you use UTF-16 internally and read/write UTF-8 files or use UTF-8 internally and convert to/from UTF-16 at the boundaries to Win32 functions? My company currently does the first, UNICODE is defined, TCHAR=UTF-16.

Re: What Every Software Developer Must Know About Unicode (2003)

#23

Post by userbinator »

lucian1900 wrote: post #17
oofabz wrote: post #11

Earlier quoted context omitted.

I think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties. UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, beca…

There is absolutely no situation in which UTF-16 wins over UTF-8, because of the surrogate pairs required. That makes both encodings variable length. UTF-32 is probably what you're thinking of.

UTF-8 is variable length in that it can be anywhere from 1 to 4 bytes, while UTF-16 can either be 2 or 4. That makes a UTF-16 decoder/encoder half as complex as a UTF-8 one.

Re: What Every Software Developer Must Know About Unicode (2003)

#24

Post by userbinator »

oofabz wrote: post #11
Scorponok wrote: post #8

Earlier quoted context omitted.

Depends what you mean by "better". UTF-8 generally ends up using fewer bytes to represent the same string than UTF-16, unless you're using certain characters a lot (e.g. for asian languages), so it's a candidate, but it's not like you could just flip a switch and make all javascript use UTF-8.

I think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties. UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, beca…

> Even worse, you may not even know it fails above the BMP, because those characters are so rare you might never test with them.

I don't think this is too relevant because anyone who claims to know UTF-16 should know about the surrogates. And if you are handling mostly Asian text (which is where UTF-16 is more likely to be chosen), then those high characters become a lot more common.

Re: What Every Software Developer Must Know About Unicode (2003)

#25

Post by lambda »

ddebernardy wrote: post #13
gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

Agreed with what you wrote except "the most backwards compatible (can be passed through many tools intended for ASCII-only)"... That amounts to knowingly sweep bugs under the rug. If a tool is intended for ascii-only, don't pass it utf-8 strings. Else, you'll probably expose yourself to malformed utf-8 strings and potential problems (e.g. php's mysql_escape vs mysql_real_escape).

Technically, it's not that it can be passed through tools intended for ASCII only. It can be passed through tools which assume an ASCII compatible character set (character set for which ASCII is a subset), which are 8 bit clean, and which don't make incorrect assumptions about being able to truncate strings at arbitrary points and be left with two valid strings.

Which is actually generally true of any tools which had been internationalized with legacy, pre-Unicode character sets like the ISO 8859 series.

Re: What Every Software Developer Must Know About Unicode (2003)

#26

Post by gilgoomesh »

asifsignupfor wrote: post #22
gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

So when writing Windows applications with Win32, would you use UTF-16 internally and read/write UTF-8 files or use UTF-8 internally and convert to/from UTF-16 at the boundaries to Win32 functions? My company currently does the first, UNICODE is defined, TCHAR=UTF-16.

I would never use anything except UTF-8 for a general text file – for the efficiency reasons listed above – unless there was a strong reason otherwise. General text files are the biggest source of problems because they have no metadata indicating what encoding they actually contain.

For everything else, it really depends what your data is for.

If your data is only ever going to hold a file path that you need to pass to the Winapi, then there's no real problem with UTF-16. Although needing to have multiple paths for your text handling can become an issue.

I write multi-platform C++ programs and even on Windows I use UTF-8 for all internal strings – including those that will eventually be passed to the Windows API. It just makes string handling simpler.

However, I use a range of abstraction classes around all OS calls that transparently converts to/from UTF-16 as needed. A good example is boost::filesystem for paths and file I/O which internally stores UTF-16 on Windows but abstracts the need for me as the programmer to know or care about that detail – instead I can use UTF-8 everywhere and let the abstraction handle the encoding.

Re: What Every Software Developer Must Know About Unicode (2003)

#27

Post by lambda »

gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

  The Winapi (aka Win32) is the only commonly used API that 
  regularly requires something other than UTF-8
Sadly, that's not really correct. Core Foundation and Cocoa on Mac OS X use UTF-16. Qt uses UTF-16. Java uses UTF-16. JavaScript uses UTF-16. Many of these APIs have easy methods for converting to and from other encodings, and offer a certain amount of abstraction over the underlying encoding, but it still shows through in that string length and character indexing work via UTF-16 code units, not Unicode code points.

  the Mac class NSString or the C# String class use UTF-16 
  internally, you don't normally need to care what they do 
  internally since any time you access the internal 
  characters, you specify the desired encoding.
While it's true that they do abstract over the encoding somewhat, and they offer conversions, the abstraction is fairly leaky, as string length and indexing all happen in terms of UTF-16 code units.

I'm a huge fan of UTF-8, and I agree that new APIs should generally favor it, but there is a lot of legacy code that has a lot of UTF-16 assumptions baked in that you can't really say that only the Winapi uses UTF-16.

Re: What Every Software Developer Must Know About Unicode (2003)

#28

Post by AnthonyMouse »

gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

> The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article).

Why has Microsoft still not fixed this? They have the various functions [Abc] which call [Abc]A and [Abc]W depending on whether UNICODE is defined, it seems obvious that they should add [Abc]U for UTF-8 and provide a similar define to make those the typedefs for the undifferentiated functions. Even if all the functions did was convert between UTF-8 and UTF-16 and call [Abc]W it would save everyone from having to write and debug their own implementation of the same thing.

> There are a few language+environment combinations that literally can't open Unicode filenames. These include MinGW C++ which has no platform independent way of opening file streams with unicode filenames.

This seems like much the same issue. I don't know if this somehow is the fault of the C++ standard or not, but it seems like there should be a way to specify (if not for the default to be) that any C or C++ standard library functions taking a const char* or std::string should Do The Right Thing when provided with a null-terminated UTF-8 string. If the OS needs something different on the bottom then let the library do the conversion -- half the point of standard libraries is to abstract away things like that instead of making everybody futz with them all the time.

Re: What Every Software Developer Must Know About Unicode (2003)

#29

Post by kmontrose »

lambda wrote: post #27
gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 Sadly, that's not really correct. Core Foundation and Cocoa on Mac OS X use UTF-16. Qt uses UTF-16. Java uses UTF-16. JavaScript uses UTF-16. Many of these APIs have easy methods for converting to and from other encodings, and offer a certain amount of abstraction over the underlying encoding, but it still shows th…

There's a reason a ton of legacy code is UCS-2^/UTF-16 and not UTF-8.

The code is older than UTF-8.

Windows NT was first released in '93, UTF-8 didn't exist (much less have wide adoption) for most of it's development. Likewise, NeXTSTEP had an initial release in '89. Java and JavaScript (both '95) could have adopted UTF-8, but they're almost certainly running on an OS that expects UTF-16 soooo... yeah.

^UCS-2 was superceeded by UTF-16, so you'll find them used interchangeably a lot.

Re: What Every Software Developer Must Know About Unicode (2003)

#30

Post by hobs »

pygy_ wrote: post #2

A good summary, but for one imortant detail: In UTF-16, some code points (laying on the so-called "astral planes", ie not on the "basic multilingual plane") take 32 bits. The Emoji, for example, lie on the first higher plane: 🍒🎄🐰🚴. Firefox and Safari display them properly, Chrome doesn't, no idea for IE and Opera. UCS-2 is a strict 16-bit encoding (a subset of UTF-16), and it cannot represent all characters. It i…

🚴 isnt displayed correctly in my Firefox, though all the other characters you mention are.

FF 26 on Win 7.

Post reply on HN