Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

61–70 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#61

Post by lubutu »

erichocean wrote: post #55

The strangest thing about Unicode (any flavor) is that NULL, aka \0, aka "all zeros" is a valid character. If you claim to support Unicode, you have to support NULL characters; otherwise, you support a subset. I find most OS utilities that "accept" Unicode fail to accept the NULL character. FWIW, UTF-8 has a few invalid characters (characters that can never appear in a valid UTF-8 string). Any one of them could be us…

No. NUL is backwards-compatible with ASCII, and is used everywhere. Choosing some arbitrary invalid UTF-8 byte for use as a terminator would be a terrible decision. If you want to handle NUL, simply use length-annotated slices instead of C-style NUL-terminated strings. Anything else is completely wrong.

Re: The UTF-8-Everywhere Manifesto

#62

Post by makecheck »

CJefferson wrote: post #33

I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness? In particular, what should I watch out for to make an ASCII parser UTF-8 clean?

If you're reading something in pieces, like a buffer that fills 256 bytes at a time, you have to be careful. UTF-8 is a multi-byte encoding so the last byte in your buffer may not completely finish a code point. Unlike older code that can just read a bunch of bytes and use them, with multi-byte encodings you have to have a way to deal with "left-overs" until new bytes show up.

Fortunately the UTF-8 encoding (e.g. see the Wikipedia page) makes it clear when a byte is the beginning of a new point and it tells you how many intermediate bytes should follow.

Re: The UTF-8-Everywhere Manifesto

#63

Post by adobriyan »

erichocean wrote: post #59
colanderman wrote: post #58

Earlier quoted context omitted.

-1 is not a valid Unicode code point. "All 1s" is not adequately defined without saying how many 1s – and Unicode does not specify a maximum bit width. Even if you said "the maximum Unicode code point", that is not all 1s – it is 0x10FFFF.

That's the entire point of choosing -1 as an "end of sequence" marker for a UTF-8 string when the length is not known up front. A byte containing all 1s is not valid in any Unicode encoding, so if one appears, you'd know you had hit the end of the string.

This doesn't work well with handling ill-formed sequences.

The length is of course known upfront if not of the whole string but at least of the individual small substring.

Re: The UTF-8-Everywhere Manifesto

#64

Post by pcwalton »

jacobolus wrote: post #36
pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

As long as you don’t care about errors, converting to UTF-8 is quite fast. Just use native calls: var decode = function (bytes) { return decodeURIComponent(escape(bytes)); } var encode = function (string) { return unescape(encodeURIComponent(string)); } (Definitely test it before wailing about benchmarks. My guess is that whatever else you’re doing is likely much slower.) If you do care about errors, or especially if…

I think you misunderstand -- I'm referring to the actual systems-level implementation of the browser engine itself. I'm not talking about the implementation of web apps.

Consider a pattern like this: A page calls document.createElement(), adds a large text node (say, the collected works of Shakespeare in text form) to it, calls window.getComputedStyle() on that element, then throws the element away. This series of DOM manipulations must go through the layout engine. If the layout engine knows only UTF-8, then the layout engine has to convert the collected works of William Shakespeare from UTF-16 to UTF-8 for no reason (as it needs an up-to-date DOM to perform CSS selector matching for the getComputedStyle() call). There is no reason to do that when it could just use UTF-16 instead and save itself the trouble.

Re: The UTF-8-Everywhere Manifesto

#65

Post by erichocean »

lubutu wrote: post #61
erichocean wrote: post #55

The strangest thing about Unicode (any flavor) is that NULL, aka \0, aka "all zeros" is a valid character. If you claim to support Unicode, you have to support NULL characters; otherwise, you support a subset. I find most OS utilities that "accept" Unicode fail to accept the NULL character. FWIW, UTF-8 has a few invalid characters (characters that can never appear in a valid UTF-8 string). Any one of them could be us…

No. NUL is backwards-compatible with ASCII, and is used everywhere. Choosing some arbitrary invalid UTF-8 byte for use as a terminator would be a terrible decision. If you want to handle NUL, simply use length-annotated slices instead of C-style NUL-terminated strings. Anything else is completely wrong.

Did you even read my comment? NULL is a valid UTF-8 character.

If specific languages and their standard libraries choose to treat it as a string terminator (C, I'm looking at you), well, then fine.

But it's still valid Unicode. If you claim to support Unicode strings, but don't support the NULL character in those strings, you don't support Unicode strings in their entirety.

Going further, there's nothing in the ASCII spec that requires NULL to only appear at the end of a valid string. That's a C language convention, AFAIK (maybe it started earlier...).

Re: The UTF-8-Everywhere Manifesto

#66

Post by adobriyan »

4ad wrote: post #60
makecheck wrote: post #57

Earlier quoted context omitted.

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

cat a b c There you go, a BOM in the middle of the file.

Unicode-compatible cat(1) would strip BOM from all but the first file.

Applying non-Unicode compatible utility to Unicode files of course doesn't work.

Re: The UTF-8-Everywhere Manifesto

#67

Post by lubutu »

erichocean wrote: post #65
lubutu wrote: post #61

Earlier quoted context omitted.

No. NUL is backwards-compatible with ASCII, and is used everywhere. Choosing some arbitrary invalid UTF-8 byte for use as a terminator would be a terrible decision. If you want to handle NUL, simply use length-annotated slices instead of C-style NUL-terminated strings. Anything else is completely wrong.

Did you even read my comment? NULL is a valid UTF-8 character. If specific languages and their standard libraries choose to treat it as a string terminator (C, I'm looking at you), well, then fine. But it's still valid Unicode. If you claim to support Unicode strings, but don't support the NULL character in those strings, you don't support Unicode strings in their entirety. Going further, there's nothing in the ASCII…

I'm sorry, did you read mine? "If you want to handle NUL, simply use length-annotated slices instead of C-style NUL-terminated strings."

Seriously though, using 0xFF as a UTF-8 string terminator would be a terrible mistake.

Re: The UTF-8-Everywhere Manifesto

#68

Post by bobbydavid »

makecheck wrote: post #57
luriel wrote: post #31

Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

> Q: What do you think about BOMs?

> A: Another reason not to use UTF-16. UTF-8 has a BOM too, even though byte order is not an issue in this encoding. This is to manifest that this is a UTF-8 stream. If UTF-8 remains the only popular encoding (as it already is in the internet world), the BOM becomes redundant. In practice, many UTF-8 text files omit BOMs today.

That's one of the central points of the article --- "sniffing" is completely unnecessary if you can assume a UTF-8 encoding type.

You're right that the mark isn't useless, we want it to be useless.

Re: The UTF-8-Everywhere Manifesto

#69

Post by erichocean »

erichocean wrote: post #65
lubutu wrote: post #61

Earlier quoted context omitted.

No. NUL is backwards-compatible with ASCII, and is used everywhere. Choosing some arbitrary invalid UTF-8 byte for use as a terminator would be a terrible decision. If you want to handle NUL, simply use length-annotated slices instead of C-style NUL-terminated strings. Anything else is completely wrong.

Did you even read my comment? NULL is a valid UTF-8 character. If specific languages and their standard libraries choose to treat it as a string terminator (C, I'm looking at you), well, then fine. But it's still valid Unicode. If you claim to support Unicode strings, but don't support the NULL character in those strings, you don't support Unicode strings in their entirety. Going further, there's nothing in the ASCII…

UPDATE: Yes, I read yours. I suggested a non-NULL terminator for strings that are not length-terminated. Your alternative only applies to length-terminated strings.

You manage this by having the sending end slice things into lengths for the receiver. Is there some globally-recognized standard for doing so that I'm not aware of?

Because if there isn't, my termination proposal (an invalid Unicode byte) is just as valid as your framing protocol.

Re: The UTF-8-Everywhere Manifesto

#70

Post by pcwalton »

ori_b wrote: post #56
pcwalton wrote: post #27

Earlier quoted context omitted.

Well, we're talking about DOM manipulation performance here. Pages that use DOM manipulation heavily will see a potentially-unacceptable performance loss if text always has to be converted to UTF-8. Is fast DOM manipulation important? Given that the only way for the sole scripting language on the Web to display anything or interact with the user is through DOM manipulation, I think it's worth optimizing every cycle..…

http://www.utf8everywhere.org/#faq.cvt.perf If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.

See my comment above; I can construct cases that make this sort of conversion have unacceptable overhead.

Would it matter in a real-world setting? I can't say for sure, because nobody I know of has tried making a production-quality UTF-8 web layout engine. But, in my mind, none of the benefits of UTF-8 (memory usage being the main one in a browser [1]) outweigh the performance risks of doing conversion. And the risk is real.

[1]: Note that you still need UTF-16 anyway, for interoperability with JavaScript. So using UTF-8 might even lead to worse memory usage, due to the necessity of duplicating strings, than a careful UTF-16-everywhere scheme that takes advantage of string buffer sharing between the JS heap and the layout engine heap would.

Post reply on HN