Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

91–100 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#91

Post by khuey »

JoshTriplett wrote: post #88
pcwalton wrote: post #64

Earlier quoted context omitted.

I think you misunderstand -- I'm referring to the actual systems-level implementation of the browser engine itself. I'm not talking about the implementation of web apps. Consider a pattern like this: A page calls document.createElement(), adds a large text node (say, the collected works of Shakespeare in text form) to it, calls window.getComputedStyle() on that element, then throws the element away. This series of DO…

What part of that process requires UTF-16? JavaScript doesn't require UTF-16; it just requires Unicode. You could use UTF-8 in your JavaScript implementation as well.

JS does require UTF-16, because the surrogate pairs of non-BMP characters are separable in JS strings.

''.length == 2 ''[0] == first codepoint in the surrogate pair ''[1] == second codepoint in the surrogate pair

Any JS implementation using UTF-8 would have to convert to UTF-16 for proper answers to .length and array indexing on strings.

Re: The UTF-8-Everywhere Manifesto

#92

Post by pkmays »

JoshTriplett wrote: post #88
pcwalton wrote: post #64

Earlier quoted context omitted.

I think you misunderstand -- I'm referring to the actual systems-level implementation of the browser engine itself. I'm not talking about the implementation of web apps. Consider a pattern like this: A page calls document.createElement(), adds a large text node (say, the collected works of Shakespeare in text form) to it, calls window.getComputedStyle() on that element, then throws the element away. This series of DO…

What part of that process requires UTF-16? JavaScript doesn't require UTF-16; it just requires Unicode. You could use UTF-8 in your JavaScript implementation as well.

> What part of that process requires UTF-16? JavaScript doesn't require UTF-16; it just requires Unicode. You could use UTF-8 in your JavaScript implementation as well.

Actually, JavaScript _does_ require UTF-16. From the ES5.1 spec:

> A conforming implementation of this Standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it presumed to be the UTF-16 encoding form.

Re: The UTF-8-Everywhere Manifesto

#93

Post by khuey »

ori_b wrote: post #56
pcwalton wrote: post #27

Earlier quoted context omitted.

Well, we're talking about DOM manipulation performance here. Pages that use DOM manipulation heavily will see a potentially-unacceptable performance loss if text always has to be converted to UTF-8. Is fast DOM manipulation important? Given that the only way for the sole scripting language on the Web to display anything or interact with the user is through DOM manipulation, I think it's worth optimizing every cycle..…

http://www.utf8everywhere.org/#faq.cvt.perf If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.

UTF-8 -> UTF-16 conversion was at one point a noticeable fraction of Firefox's startup time.

https://bugzilla.mozilla.org/show_bug.cgi?id=506431

Since then we've done things such as fast-path ASCII -> UTF-16 conversion with SSE2 instructions. Converting a few dozen bytes is unlikely to make a significant difference, but often one needs to deal with more than a few dozen bytes.

Re: The UTF-8-Everywhere Manifesto

#94

Post by erichocean »

lubutu wrote: post #46
makecheck wrote: post #30

Markus Kuhn's web page has a lot of useful UTF-8 info and valuable links (e.g. samples of UTF-8 corner cases that people often miss). http://www.cl.cam.ac.uk/~mgk25/unicode.html

This is a great resource; it was extremely useful when I was writing a UTF-8 library myself. I found the UTF-8 stress test file is particularly useful to run tests against: http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt

Hmm, TextMate has problems with "5.2 Paired UTF-16 surrogates" in that stress test file.

(Yes, I interpreted the file as UTF-8 in TextMate).

Re: The UTF-8-Everywhere Manifesto

#95

Post by ruediger »

lambda wrote: post #38
pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

Yeah, it's really sad the number of legacy APIs which have standardized on UTF-16. The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it. I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text…

ICU has some UTF-8 functionality. But ICU is really ugly.

http://userguide.icu-project.org/strings/utf-8

Re: The UTF-8-Everywhere Manifesto

#96

Post by scoith »

That page is misleading when it comes to Japanese text: UTF-8 sucks for Japanese text. UTF-8 and UTF-16 aren't the only two choices within the whole world, which is demonstrated in their choice of encoding Shift-JIS.

Re: The UTF-8-Everywhere Manifesto

#97

Post by ruediger »

mkup wrote: post #72

I use UTF-8 for transmitted data and disk I/O, and I use UCS-4 (wchar_t on Linux/FreeBSD) for internal representation of strings in my software. I generally agree with this article, but I disagree with it on the point that UTF-8 is the only appropriate encoding for strings stored in memory, and also I disagree on the point wchar_t should be removed from C++ standard or made sizeof 1, as in Android NDK. Let me explain…

> Also UTF-8 string can't be cut at arbitrary position.

Neither can be any other kind of Unicode string because of Combining Characters. That's why the Unicode standard (or an Annex) recommends algorithms for text segmentation.

(And if you really need to cut at a certain length then you can easily backtrack and find the beginning of the sequence by looking for the first byte with the MSB = 0)

Re: The UTF-8-Everywhere Manifesto

#98

Post by ruediger »

scoith wrote: post #96

That page is misleading when it comes to Japanese text: UTF-8 sucks for Japanese text. UTF-8 and UTF-16 aren't the only two choices within the whole world, which is demonstrated in their choice of encoding Shift-JIS.

Can you elaborate on that? Why does Unicode suck for Japanese text?

Re: The UTF-8-Everywhere Manifesto

#99

Post by JoshTriplett »

khuey wrote: post #91
JoshTriplett wrote: post #88

Earlier quoted context omitted.

What part of that process requires UTF-16? JavaScript doesn't require UTF-16; it just requires Unicode. You could use UTF-8 in your JavaScript implementation as well.

JS does require UTF-16, because the surrogate pairs of non-BMP characters are separable in JS strings. ' '.length == 2 ' '[0] == first codepoint in the surrogate pair ' '[1] == second codepoint in the surrogate pair Any JS implementation using UTF-8 would have to convert to UTF-16 for proper answers to .length and array indexing on strings.

Still doesn't mean that it has to store the strings in UTF-16. And a bit of common-case optimizing would allow ignoring that case until a string actually ends up with a non-BMP character in it.

Alternatively, if a JavaScript implementation chose to completely ignore that particular requirement, I'd guess that approximately zero pages (outside of test cases) would break, and a few currently broken pages (that assumed sane Unicode handling) would start working.

Re: The UTF-8-Everywhere Manifesto

#100

Post by ajross »

makecheck wrote: post #57
luriel wrote: post #31

Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

What on earth does "processed as such" mean? UTF-8 can be "processed" anywhere ASCII can, that's the whole point. The only point to the BOM is to distinguish it from UTF-16 (or UCS2, which is usually what UTF-16 degenerates into). And UTF-16 is broken garbage and shouldn't be used.

Given that, your last sentence is basically an ode to complexity. Taken to the logical conclusion you'd support any rule, no matter how ridiculous, as long as it is agreed upon as "correct" by ... someone.

But I don't agree: the BOM hurts, it doesn't help.

Post reply on HN