The UTF-8-Everywhere Manifesto • Hacker News Board

Re: The UTF-8-Everywhere Manifesto

#111

Post by cobbal » Sun, Apr 29, 2012, 9:59 PM UTC

__david__ wrote: post #51

natch wrote: post #40

Strings (NSString) on Apple platforms are UTF-16. The Apple platforms are not exactly lagging behind in either multilingual, or text processing. I wonder what this team of three people knows that Apple doesn't? Or is it the other way around, that Apple knows something they don't, and when it comes to shipping products that work in the real world, Apple has figured out how to do it?

NSStrings are opaque--you always call accessor functions and never have access to the low level backing store. The reason they are good is that you can't get data into or out of them without specifying an encoding, which leaves the actual encoding of the backing store as an implementation detail. The fact is, I don't even know (or see documented) that the backing is UTF-16--Apple is free to change that at their whim…

Unfortunately, this isn't entirely true.

  - (unichar)characterAtIndex:(NSUInteger)index

will return the UTF-16 "characters" (codepoints or half-codepoints)

Re: The UTF-8-Everywhere Manifesto

#112

Post by petercooper » Sun, Apr 29, 2012, 10:00 PM UTC

pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

A recent discussion about this involving Brendan Eich and a few other JS folk: https://gist.github.com/1850768

Re: The UTF-8-Everywhere Manifesto

#113

Post by lmm » Sun, Apr 29, 2012, 10:12 PM UTC

haberman wrote: post #54

Totally agree re: UTF-8 vs other Unicode encodings. But are there still still hold-outs who don't like Unicode? Last I heard some CJK users were unhappy about Han Unification: http://en.wikipedia.org/wiki/Han_unification

The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

Re: The UTF-8-Everywhere Manifesto

#114

Post by soc88 » Sun, Apr 29, 2012, 10:29 PM UTC

pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

Claims like this are exactly the reason why we will still be stuck with multiple useless encodings in 2030.

I wonder how long it will take until people find their balls and decide to move towards the right direction.

Re: The UTF-8-Everywhere Manifesto

#115

Post by soc88 » Sun, Apr 29, 2012, 10:33 PM UTC

khuey wrote: post #91

JoshTriplett wrote: post #88

Earlier quoted context omitted.

What part of that process requires UTF-16? JavaScript doesn't require UTF-16; it just requires Unicode. You could use UTF-8 in your JavaScript implementation as well.

JS does require UTF-16, because the surrogate pairs of non-BMP characters are separable in JS strings. ' '.length == 2 ' '[0] == first codepoint in the surrogate pair ' '[1] == second codepoint in the surrogate pair Any JS implementation using UTF-8 would have to convert to UTF-16 for proper answers to .length and array indexing on strings.

If stuff like length and indexing depend on the actual underlying representation, the implementation is broken regardless of the actual format used.

Re: The UTF-8-Everywhere Manifesto

#116

Post by lambda » Sun, Apr 29, 2012, 10:37 PM UTC

brigade wrote: post #106

lambda wrote: post #38

Earlier quoted context omitted.

Yeah, it's really sad the number of legacy APIs which have standardized on UTF-16. The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it. I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text…

OS X APIs generally use NSString/CFString, which hide the actual encoding of the string; they can be any encoding at all internally.

While they could, in theory, hide the actual encoding, the APIs all refer to UTF-16 code units as "characters;" so while they could use UTF-8 as the internal encoding, you need to use and understand UTF-16 in order to interact with them properly. When you ask for the "length" of a string, you are told the number of UTF-16 code units. When you get a character at an index, you get the UTF-16 code unit. That's what I mean when I say the APIs use UTF-16; everything in the API that deals with individual "chracters" is actually referring to UTF-16 code units.

The same is true of JavaScript; while you could technically implement the strings however you want, the APIs are all oriented around UTF-16 code units. And the Windows API as well, is all built around UTF-16 code units.

The problem with all of these APIs is that they make the mistake of conflating characters and code units. They all make the assumption that a character consists of a single, fixed width integer, of some given size (16 bits in the case of UTF-16). It is better to distinguish between indexing in code units (such as bytes in UTF-8 or 16 bit integers in UTF-16) and indexing in code points, or glyphs, or whatever higher level concept you are talking about. Really, for anything higher than the code unit level, you should be dealing with variable-length strings, and not try to force that into fixed length units. With UTF-8, there's no temptation to treat a single code unit as being an independently meaningful entity, as that assumption breaks down as soon as you get past the ASCII range; while with UTF-16, it's easy to make that mistake, since it holds true for everything in the Basic Multilingual Plane, which contains most characters you're likely to encounter on a day to day basis.

Re: The UTF-8-Everywhere Manifesto

#117

Post by sanxiyn » Sun, Apr 29, 2012, 10:37 PM UTC

haberman wrote: post #54

Totally agree re: UTF-8 vs other Unicode encodings. But are there still still hold-outs who don't like Unicode? Last I heard some CJK users were unhappy about Han Unification: http://en.wikipedia.org/wiki/Han_unification

I am a Korean user (K in CJK), and no one, I repeat, no one, care about Han unification here.

I heard that it is different in China and Japan though.

Re: The UTF-8-Everywhere Manifesto

#118

Post by sanxiyn » Sun, Apr 29, 2012, 10:42 PM UTC

lmm wrote: post #113

haberman wrote: post #54

Totally agree re: UTF-8 vs other Unicode encodings. But are there still still hold-outs who don't like Unicode? Last I heard some CJK users were unhappy about Han Unification: http://en.wikipedia.org/wiki/Han_unification

The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

This is false. UTF-8 sorts Korean almost correctly. For practical purposes, you can use sort-by-unicode-codepoint to sort Korean.

Re: The UTF-8-Everywhere Manifesto

#119

Post by btilly » Sun, Apr 29, 2012, 10:44 PM UTC

soc88 wrote: post #115

khuey wrote: post #91

Earlier quoted context omitted.

JS does require UTF-16, because the surrogate pairs of non-BMP characters are separable in JS strings. ' '.length == 2 ' '[0] == first codepoint in the surrogate pair ' '[1] == second codepoint in the surrogate pair Any JS implementation using UTF-8 would have to convert to UTF-16 for proper answers to .length and array indexing on strings.

If stuff like length and indexing depend on the actual underlying representation, the implementation is broken regardless of the actual format used.

Change "implementation" to spec and I agree with you.

An implementation that fails to meet the spec can be argued as not broken, even if said spec is broken.

Re: The UTF-8-Everywhere Manifesto

#120

Post by unconed » Sun, Apr 29, 2012, 10:44 PM UTC

makecheck wrote: post #78

unconed wrote: post #73

Earlier quoted context omitted.

This argument is silly. Why not prefix every UTF-8 string with a BOM then? It's wasteful and unnecessary, because UTF-8's clean structure already makes it trivial to detect, and false positives are all but impossible for real-world text. There's a paper out there that proves this. The UTF-8 BOM was a Microsoft invention. Nobody else uses it, and it breaks tons of things. Two examples off the top of my head: Unix hash…

You wouldn't prefix every string with it because presumably your API or program's state has already determined the string's encoding. I am not suggesting that every fragment of text has to be explicit (I agree that would be ridiculous). I am only stating facts: there is nothing incorrect about having the mark, a conformant reader must be able to handle the mark, and the mark has some value as a short-cut for avoiding…

The thing is, the BOM is metadata, it doesn't belong in content. It violates the contract of .txt files, which is: the entire file is a single string of content.

Recognizing it at the edges of your program and stripping it out is not the end of the world, but it's annoying and no other (8-bit) encoding works that way. In fact, I find it hard to believe UTF-8 BOMs in MS programs were anything more than a programmer error. Once such files were out in the wild, everyone else had to deal with them.