Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

51–60 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#51

Post by __david__ »

natch wrote: post #40

Strings (NSString) on Apple platforms are UTF-16. The Apple platforms are not exactly lagging behind in either multilingual, or text processing. I wonder what this team of three people knows that Apple doesn't? Or is it the other way around, that Apple knows something they don't, and when it comes to shipping products that work in the real world, Apple has figured out how to do it?

NSStrings are opaque--you always call accessor functions and never have access to the low level backing store. The reason they are good is that you can't get data into or out of them without specifying an encoding, which leaves the actual encoding of the backing store as an implementation detail.

The fact is, I don't even know (or see documented) that the backing is UTF-16--Apple is free to change that at their whim and no user programs would break.

Re: The UTF-8-Everywhere Manifesto

#52

Post by lambda »

sopooneo wrote: post #19

Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?

No. You never need to know how to interpret the bits in each byte; you cannot address individual bits. Endianness refers to how different bytes within a multi-byte value are addressed; little endian means that the smaller addresses refer to the lower order bytes, big endian mean that smaller addresses refer to the higher order bytes.

Think of it as which order you write the digits in a number. Each digit (byte) means the same thing regardless of whether you are big-endian or little-endian. But in big endian, you would write one thousand two hundred thirty four as 1234, while in little endian you would write it 4321.

Re: The UTF-8-Everywhere Manifesto

#53

Post by lambda »

sopooneo wrote: post #29
kijin wrote: post #23

Earlier quoted context omitted.

It's endianness independent in the sense that the order in which you interpret the bytes in each character does not depend on the processor architecture, unlike UTF-16. If your processor interprets the bits in each byte in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings. http://en.wikipedia.org/wiki/Endianness

Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?

Yes, basically. UTF-8 doesn't encode a code point as a single integer; it encodes it as a sequence of bytes, with a particular order, where some of the bits are used to represent the code point, and some of them are just used to represent whether you are looking at an initial byte or a continuation byte.

I'd recommend checking out the description of UTF-8 in Wikipdia. The tables make it fairly clear how the encoding works: http://en.wikipedia.org/wiki/UTF-8#Description

Re: The UTF-8-Everywhere Manifesto

#55

Post by erichocean »

The strangest thing about Unicode (any flavor) is that NULL, aka \0, aka "all zeros" is a valid character.

If you claim to support Unicode, you have to support NULL characters; otherwise, you support a subset.

I find most OS utilities that "accept" Unicode fail to accept the NULL character.

FWIW, UTF-8 has a few invalid characters (characters that can never appear in a valid UTF-8 string). Any one of them could be used as an "end of string" terminator if so desired, for situations where the string length is not known up front.

We could even standardize which one (hint hint). I suggest -1 (all 1s).

UPDATE: I meant "strange" as in "surprising", especially for those coming from a C background, like me.

Re: The UTF-8-Everywhere Manifesto

#56

Post by ori_b »

pcwalton wrote: post #27
nknight wrote: post #24

Earlier quoted context omitted.

> converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks So what? Is your goal to create useful software, or win at worthless benchmarks?

Well, we're talking about DOM manipulation performance here. Pages that use DOM manipulation heavily will see a potentially-unacceptable performance loss if text always has to be converted to UTF-8. Is fast DOM manipulation important? Given that the only way for the sole scripting language on the Web to display anything or interact with the user is through DOM manipulation, I think it's worth optimizing every cycle..…

http://www.utf8everywhere.org/#faq.cvt.perf

If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.

Re: The UTF-8-Everywhere Manifesto

#57

Post by makecheck »

luriel wrote: post #31

Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly.

Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it will screw up a lot more than the mark at the beginning of the file.

Re: The UTF-8-Everywhere Manifesto

#58

Post by colanderman »

erichocean wrote: post #55

The strangest thing about Unicode (any flavor) is that NULL, aka \0, aka "all zeros" is a valid character. If you claim to support Unicode, you have to support NULL characters; otherwise, you support a subset. I find most OS utilities that "accept" Unicode fail to accept the NULL character. FWIW, UTF-8 has a few invalid characters (characters that can never appear in a valid UTF-8 string). Any one of them could be us…

-1 is not a valid Unicode code point. "All 1s" is not adequately defined without saying how many 1s – and Unicode does not specify a maximum bit width. Even if you said "the maximum Unicode code point", that is not all 1s – it is 0x10FFFF.

Re: The UTF-8-Everywhere Manifesto

#59

Post by erichocean »

colanderman wrote: post #58
erichocean wrote: post #55

The strangest thing about Unicode (any flavor) is that NULL, aka \0, aka "all zeros" is a valid character. If you claim to support Unicode, you have to support NULL characters; otherwise, you support a subset. I find most OS utilities that "accept" Unicode fail to accept the NULL character. FWIW, UTF-8 has a few invalid characters (characters that can never appear in a valid UTF-8 string). Any one of them could be us…

-1 is not a valid Unicode code point. "All 1s" is not adequately defined without saying how many 1s – and Unicode does not specify a maximum bit width. Even if you said "the maximum Unicode code point", that is not all 1s – it is 0x10FFFF.

That's the entire point of choosing -1 as an "end of sequence" marker for a UTF-8 string when the length is not known up front.

A byte containing all 1s is not valid in any Unicode encoding, so if one appears, you'd know you had hit the end of the string.

Re: The UTF-8-Everywhere Manifesto

#60

Post by 4ad »

makecheck wrote: post #57
luriel wrote: post #31

Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

    cat a b c
There you go, a BOM in the middle of the file.
Post reply on HN