Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Every Software Developer Must Know About Unicode (2003)

joelonsoftware.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: What Every Software Developer Must Know About Unicode (2003)

#11

Post by oofabz »

Scorponok wrote: post #8
shousper wrote: post #6

Earlier quoted context omitted.

I wonder how hard it'd be to get JavaScript/ECMAScript onto a better encoding.. Do we actually have a "better" encoding?

Depends what you mean by "better". UTF-8 generally ends up using fewer bytes to represent the same string than UTF-16, unless you're using certain characters a lot (e.g. for asian languages), so it's a candidate, but it's not like you could just flip a switch and make all javascript use UTF-8.

I think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties.

UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, because those characters are so rare you might never test with them. But in UTF-8, if you screw up multi-byte characters, any non-ASCII character will trigger the bug, and you will fix your code more quickly.

Also, UTF-8 does not suffer from endianness issues like UTF-16 does. Few people use the BOM and no one likes it. And most importantly, UTF-8 is compatible with ASCII.

Re: What Every Software Developer Must Know About Unicode (2003)

#12

Post by gilgoomesh »

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should:

    Use UTF-8 everywhere you can.
UTF-8 is:

* the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs)

* most likely to give an appropriate result if the end-user incorrectly interprets it

* the most space efficient encoding (on average)

* avoids endianness problems

* de-facto encoding for most Mac and Linux C APIs

* verifiable with a high degree of accuracy (unlike many other encodings which can't be verified at all)

Specifically:

* If you have to pick an encoding, always try to use UTF-8 unless you're only storing text to pass into an API which requires something different.

* The Winapi (aka Win32) is the only commonly used API that regularly requires something other than UTF-8 (the Windows Unicode APIs use UTF-16 – not UCS-2 as indicated in the Spolsky article). Windows' UTF-16 requirement a pain for platform independence -- be careful. However, you should still aim to use UTF-8 for all text files on Windows and only use UTF-16 for the Windows API calls (never use the locale specific non-Unicode encodings).

* There are a few language+environment combinations that literally can't open Unicode filenames. These include MinGW C++ which has no platform independent way of opening file streams with unicode filenames. You need to fall back to C _wfopen and UTF-16 to open files correctly.

Note: you don't always have to choose the encoding. e.g. the Mac class NSString or the C# String class use UTF-16 internally, you don't normally need to care what they do internally since any time you access the internal characters, you specify the desired encoding. You should usually extract characters in UTF-8.

Re: What Every Software Developer Must Know About Unicode (2003)

#13

Post by ddebernardy »

gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

Agreed with what you wrote except "the most backwards compatible (can be passed through many tools intended for ASCII-only)"... That amounts to knowingly sweep bugs under the rug.

If a tool is intended for ascii-only, don't pass it utf-8 strings. Else, you'll probably expose yourself to malformed utf-8 strings and potential problems (e.g. php's mysql_escape vs mysql_real_escape).

Re: What Every Software Developer Must Know About Unicode (2003)

#14

Post by wrhs »

pygy_ wrote: post #2

A good summary, but for one imortant detail: In UTF-16, some code points (laying on the so-called "astral planes", ie not on the "basic multilingual plane") take 32 bits. The Emoji, for example, lie on the first higher plane: 🍒🎄🐰🚴. Firefox and Safari display them properly, Chrome doesn't, no idea for IE and Opera. UCS-2 is a strict 16-bit encoding (a subset of UTF-16), and it cannot represent all characters. It i…

Umm, this is like Unicode 101. Mentioning it here is just patronizing.

Re: What Every Software Developer Must Know About Unicode (2003)

#15

Post by wrhs »

oofabz wrote: post #11
Scorponok wrote: post #8

Earlier quoted context omitted.

Depends what you mean by "better". UTF-8 generally ends up using fewer bytes to represent the same string than UTF-16, unless you're using certain characters a lot (e.g. for asian languages), so it's a candidate, but it's not like you could just flip a switch and make all javascript use UTF-8.

I think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties. UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, beca…

Back in the real world though, UTF-16 almost never "wins" size comparisons in practice. For example, one would think that UTF-16 would be "better" than UTF-8 for pure Thai documents (2 bytes vs 3 bytes per Thai character) but in reality such a comparison is irrelevant, since the Thai government have decided that both are unacceptable in most cases and instead mandate the use of TIS-620 (1 byte per ASCII or Thai character).

It's a similar story in almost every other case where UTF-16 may seem like the best choice for size purposes. There's usually a specialist or national standard encoding in place, with much more traction (and often required by law), that is smaller and doesn't share any of the heinous issues suffered by UTF-16.

tl;dr: I don't think you made the point strongly enough. UTF-16 is an absolute joke and will only stay around due to Microsoft's stubbornness and lack of foresight. If you voluntarily use UTF-16 in 2014, you are a moron.

Re: What Every Software Developer Must Know About Unicode (2003)

#16

Post by wrhs »

golergka wrote: post #10
pygy_ wrote: post #2

A good summary, but for one imortant detail: In UTF-16, some code points (laying on the so-called "astral planes", ie not on the "basic multilingual plane") take 32 bits. The Emoji, for example, lie on the first higher plane: 🍒🎄🐰🚴. Firefox and Safari display them properly, Chrome doesn't, no idea for IE and Opera. UCS-2 is a strict 16-bit encoding (a subset of UTF-16), and it cannot represent all characters. It i…

Emojis are even more fun than that. Some of them take two unicode characters.

I think you mean "two UTF-16 codepoints". "Unicode" and UTF-16 are not interchangeable terms. Nor are "character" and codepoint.

Re: What Every Software Developer Must Know About Unicode (2003)

#17

Post by lucian1900 »

oofabz wrote: post #11
Scorponok wrote: post #8

Earlier quoted context omitted.

Depends what you mean by "better". UTF-8 generally ends up using fewer bytes to represent the same string than UTF-16, unless you're using certain characters a lot (e.g. for asian languages), so it's a candidate, but it's not like you could just flip a switch and make all javascript use UTF-8.

I think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties. UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, beca…

There is absolutely no situation in which UTF-16 wins over UTF-8, because of the surrogate pairs required. That makes both encodings variable length.

UTF-32 is probably what you're thinking of.

Re: What Every Software Developer Must Know About Unicode (2003)

#18

Post by gilgoomesh »

ddebernardy wrote: post #13
gilgoomesh wrote: post #12

This article deals mostly with Windows and is from 2003 so it fails to emphasise the current standard practice as much as it should: Use UTF-8 everywhere you can. UTF-8 is: * the most backwards compatible (can be passed through many tools intended for ASCII-only with a few limitations – including avoiding composed latin glyphs) * most likely to give an appropriate result if the end-user incorrectly interprets it * th…

Agreed with what you wrote except "the most backwards compatible (can be passed through many tools intended for ASCII-only)"... That amounts to knowingly sweep bugs under the rug. If a tool is intended for ascii-only, don't pass it utf-8 strings. Else, you'll probably expose yourself to malformed utf-8 strings and potential problems (e.g. php's mysql_escape vs mysql_real_escape).

Yes, obviously there will be caveats when using a tool beyond its intended domain:

* you do need to know that the tool passes non-ASCII through unchanged

* the text should not contain composed latin glyphs

* you're on your own if you're trimming strings to byte lengths

I've added the second point to my comment.

It's not about sweeping bugs under the rug at all. It's about using non-latin text on the command-line and in code. Most command-line tools are ASCII but will pass through non-ASCII characters unchanged. However, most require that you avoid composed latin glyphs. Since Unicode includes single-codepoint versions of all valid latin accented glyphs and these are the default entry methods, this isn't usually a problem but yes, it really constitutes a subset of UTF-8 and you must know about this limitation to avoid bugs.

Re: What Every Software Developer Must Know About Unicode (2003)

#19

Post by oofabz »

lucian1900 wrote: post #17
oofabz wrote: post #11

Earlier quoted context omitted.

I think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties. UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, beca…

There is absolutely no situation in which UTF-16 wins over UTF-8, because of the surrogate pairs required. That makes both encodings variable length. UTF-32 is probably what you're thinking of.

I know that both encodings are variable-length. That is the issue I am trying to address.

My point is that in UTF-16 it's too easy to ignore surrogate pairs. Lots of UTF-16 software fails to handle variable-length characters because they are so rare. But in UTF-8 you can't ignore multi-byte characters without obvious bugs. These bugs are noticed and fixed more quickly than UTF-16 surrogate pair bugs. This makes UTF-8 more reliable.

I am not sure why you think I am advocating UTF-16. I said almost nothing good about it.

Re: What Every Software Developer Must Know About Unicode (2003)

#20

Post by millstone »

oofabz wrote: post #11
Scorponok wrote: post #8

Earlier quoted context omitted.

Depends what you mean by "better". UTF-8 generally ends up using fewer bytes to represent the same string than UTF-16, unless you're using certain characters a lot (e.g. for asian languages), so it's a candidate, but it's not like you could just flip a switch and make all javascript use UTF-8.

I think the size issue is a red herring. UTF-8 wins some, UTF-16 wins others, but either encoding is acceptable. There is no clear winner here so we should look at other properties. UTF-8 is more reliable, because mishandling variable-length characters is more obvious. In UTF-16 it's easy to write something that works with the BMP and call it good enough. Even worse, you may not even know it fails above the BMP, beca…

UTF-8 has its own unique issues, like non-shortest forms and invalid code units, that you are even less likely to encounter in the wild. Bugs in handling of these have enabled security exploits in the past.
Post reply on HN