The UTF-8-Everywhere Manifesto • Hacker News Board

Re: The UTF-8-Everywhere Manifesto

#41

Post by ori_b » Sun, Apr 29, 2012, 7:03 PM UTC

sopooneo wrote: post #29

kijin wrote: post #23

Earlier quoted context omitted.

It's endianness independent in the sense that the order in which you interpret the bytes in each character does not depend on the processor architecture, unlike UTF-16. If your processor interprets the bits in each byte in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings. http://en.wikipedia.org/wiki/Endianness

Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?

utf-8 is a single byte encoding. Reversing the order of a sequence that's one byte long just gives back that one byte.

Re: The UTF-8-Everywhere Manifesto

#42

Post by dlubarov » Sun, Apr 29, 2012, 7:05 PM UTC

CJefferson wrote: post #33

I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness? In particular, what should I watch out for to make an ASCII parser UTF-8 clean?

All 7-bit ASCII is valid UTF-8, so you're fine as long as you're really using 7-bit ASCII and not latin1 or Windows-1252, etc.

Re: The UTF-8-Everywhere Manifesto

#43

Post by antidoh » Sun, Apr 29, 2012, 7:06 PM UTC

Text is maddening, the modern Tower of Babel.

Is there a definitive reference, or small handful of references, to learn all that's worth knowing about text, from ASCII to UTF-∞ and beyond?

Re: The UTF-8-Everywhere Manifesto

#44

Post by [deleted] » Sun, Apr 29, 2012, 7:06 PM UTC

natch wrote: post #40

Strings (NSString) on Apple platforms are UTF-16. The Apple platforms are not exactly lagging behind in either multilingual, or text processing. I wonder what this team of three people knows that Apple doesn't? Or is it the other way around, that Apple knows something they don't, and when it comes to shipping products that work in the real world, Apple has figured out how to do it?

[deleted]

Re: The UTF-8-Everywhere Manifesto

#45

Post by Xuzz » Sun, Apr 29, 2012, 7:06 PM UTC

natch wrote: post #40

Strings (NSString) on Apple platforms are UTF-16. The Apple platforms are not exactly lagging behind in either multilingual, or text processing. I wonder what this team of three people knows that Apple doesn't? Or is it the other way around, that Apple knows something they don't, and when it comes to shipping products that work in the real world, Apple has figured out how to do it?

NSString is decades old, from NeXTSTEP (you can see that in the name: "NS"). While its possible they could change it, the in-memory representation doesn't matter much in this case. When you transfer data out, such as with writeToFile:encoding: or convert it into a char * (often with UTF8String, or one of the C string methods), you are almost always specifying an encoding anyway. And, for most Cocoa apps, that encoding is UTF8.

Re: The UTF-8-Everywhere Manifesto

#46

Post by lubutu » Sun, Apr 29, 2012, 7:09 PM UTC

makecheck wrote: post #30

Markus Kuhn's web page has a lot of useful UTF-8 info and valuable links (e.g. samples of UTF-8 corner cases that people often miss). http://www.cl.cam.ac.uk/~mgk25/unicode.html

This is a great resource; it was extremely useful when I was writing a UTF-8 library myself. I found the UTF-8 stress test file is particularly useful to run tests against: http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt

Re: The UTF-8-Everywhere Manifesto

#47

Post by gregschlom » Sun, Apr 29, 2012, 7:09 PM UTC

natch wrote: post #40

Strings (NSString) on Apple platforms are UTF-16. The Apple platforms are not exactly lagging behind in either multilingual, or text processing. I wonder what this team of three people knows that Apple doesn't? Or is it the other way around, that Apple knows something they don't, and when it comes to shipping products that work in the real world, Apple has figured out how to do it?

As the authors explain in the post, many things use UTF-16 internally (Python, Java, C#, etc...). It does not mean that it's the best solution.

Have you read the article? They make a good explanation at why UTF-16 is "the worst of both worlds" (wide characters AND variable lenght).

Re: The UTF-8-Everywhere Manifesto

#48

Post by rogerbinns » Sun, Apr 29, 2012, 7:10 PM UTC

CJefferson wrote: post #33

I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness? In particular, what should I watch out for to make an ASCII parser UTF-8 clean?

If you are "parsing" a string then you will have problems unless you specifically make the code deal with unicode code points and not bytes. If you just accept a char* and then pass it on with the contents as is you'll generally be fine (except on Windows).

Re: The UTF-8-Everywhere Manifesto

#49

Post by antidoh » Sun, Apr 29, 2012, 7:10 PM UTC

lambda wrote: post #38

pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

Yeah, it's really sad the number of legacy APIs which have standardized on UTF-16. The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it. I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text…

Some old languages have also made that choice, albeit recently in Python's case:

http://www.python.org/dev/peps/pep-3120/

Re: The UTF-8-Everywhere Manifesto

#50

Post by ori_b » Sun, Apr 29, 2012, 7:10 PM UTC

maaku wrote: post #35

paulsutter wrote: post #15

Earlier quoted context omitted.

The author makes a compelling case for UTF-8 in Asian languages. I'd love to hear any specific counter-arguments.

No he doesn't, he dismisses it out of hand by choosing an example that is uniquely suited to minimize the advantages of UTF-16 for non-Roman scripts. Precious little of an HTML document is actually textual content.

I think that's the point; In the real world, any significant chunk of non-roman text is embedded in far more roman text, or in terms of internals of programs, is generally dwarfed by the size of other data structures.

More or less, I'd say that storage size of text usually doesn't matter.