Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

21–30 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#21

Post by pcwalton »

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks.

For new APIs in which legacy interoperability isn't needed, I completely approve of this document.

Re: The UTF-8-Everywhere Manifesto

#22

Post by evincarofautumn »

sopooneo wrote: post #19

Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?

No, that’s not endianness; endianness refers to the ordering of bytes within a multi-byte value—least significant byte first or most significant byte first, generally. The order of octets in a UTF-8 code point is fixed, and because a bit is not an addressable unit of memory, the storage order of bits within an octet is immaterial.

Re: The UTF-8-Everywhere Manifesto

#23

Post by kijin »

sopooneo wrote: post #19

Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?

It's endianness independent in the sense that the order in which you interpret the bytes in each character does not depend on the processor architecture, unlike UTF-16.

If your processor interprets the bits in each byte in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings.

http://en.wikipedia.org/wiki/Endianness

Re: The UTF-8-Everywhere Manifesto

#24

Post by nknight »

pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

> converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks

So what? Is your goal to create useful software, or win at worthless benchmarks?

Re: The UTF-8-Everywhere Manifesto

#25

Post by gavanwoolery »

pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

I agree with you in concept, but the Web is actually not strictly tied to javascript. In the short term (say, next decade), JS is probably not going anywhere, but in the longer term I hope someone creates a more well-thought-out replacement. (And for the record, I kind of like javascript, just a few things I would change with it.)

Re: The UTF-8-Everywhere Manifesto

#26

Post by raverbashing »

lambda wrote: post #16
raverbashing wrote: post #3

Disagree "UTF-16 is the worst of both worlds—variable length and too wide" Really, the author tries to convince the reader, but it's not that clean cut. One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32) "It's not fix…

> One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32) First of all, if you don't know the encoding, then you don't know the encoding, and you will need to figure out if it's UTF-8, UTF-16, ISO-8859-1, etc. If you happe…

"First of all, if you don't know the encoding, then you don't know the encoding"

True. But as you said, you have to know if it's BE or LE on UTF16. And there are ways to determine that automatically. (or it's on the same platform so it doesn't matter). With "ASCII compatible" codes, you can't.

I guess the main issue to me is that UTF-16 is not "ASCII compatible" so you know it's a different beast altogether.

And don't worry, I'm not assuming UTF-16 is fixed width. One should use the libraries and not try to solve this 'manually'.

About UTF-32 think: CPU registers and operations. Working with bytes is inefficient (even with the benefit of smaller size).

Re: The UTF-8-Everywhere Manifesto

#27

Post by pcwalton »

nknight wrote: post #24
pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

> converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks So what? Is your goal to create useful software, or win at worthless benchmarks?

Well, we're talking about DOM manipulation performance here. Pages that use DOM manipulation heavily will see a potentially-unacceptable performance loss if text always has to be converted to UTF-8.

Is fast DOM manipulation important? Given that the only way for the sole scripting language on the Web to display anything or interact with the user is through DOM manipulation, I think it's worth optimizing every cycle...

Re: The UTF-8-Everywhere Manifesto

#29

Post by sopooneo »

kijin wrote: post #23
sopooneo wrote: post #19

Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?

It's endianness independent in the sense that the order in which you interpret the bytes in each character does not depend on the processor architecture, unlike UTF-16. If your processor interprets the bits in each byte in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings. http://en.wikipedia.org/wiki/Endianness

Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?
Post reply on HN