The UTF-8-Everywhere Manifesto • Hacker News Board

Re: The UTF-8-Everywhere Manifesto

#181

Post by haberman » Mon, Apr 30, 2012, 9:02 PM UTC

lmm wrote: post #113

haberman wrote: post #54

Totally agree re: UTF-8 vs other Unicode encodings. But are there still still hold-outs who don't like Unicode? Last I heard some CJK users were unhappy about Han Unification: http://en.wikipedia.org/wiki/Han_unification

The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

Do you think that sort-by-unicode-codepoint is good enough to use for technical contexts where most content is english or at least represented by the Latin alphabet? For example, do you think it's a valid choice to sort by codepoint for Java symbol names in a code refactoring tool?

I ask because I expect that sort-by-codepoint is an order of magnitude more efficient.

Re: The UTF-8-Everywhere Manifesto

#182

Post by blasdel » Tue, May 01, 2012, 5:18 AM UTC

fleitz wrote: post #131

tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else. UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better. There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8. C#…

Java does NOT implement UTF-16, nor do most systems that claim to.

It's UCS-2, which is its own even worse pile of bullshit.

Re: The UTF-8-Everywhere Manifesto

#183

Post by ybungalobill » Tue, May 01, 2012, 7:28 AM UTC

ori_b wrote: post #172

kijin wrote: post #127

Earlier quoted context omitted.

No it isn't. Any letter with an accent will take up two bytes. Most non-Latin characters take up three bytes, sometimes even four.

Poorly phrased. It can take multiple bytes to fully define one codepoint, but the encoding is defined in terms of a stream of single bytes. In other words, each unit is one byte, hence flipping each unit gives back the same unit. This is not the case for UTF-16 and UTF-32.

AFAIK the correct term is "byte oriented".

Re: The UTF-8-Everywhere Manifesto

#184

Post by lmm » Tue, May 01, 2012, 1:02 PM UTC

haberman wrote: post #181

lmm wrote: post #113

Earlier quoted context omitted.

The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

Do you think that sort-by-unicode-codepoint is good enough to use for technical contexts where most content is english or at least represented by the Latin alphabet? For example, do you think it's a valid choice to sort by codepoint for Java symbol names in a code refactoring tool? I ask because I expect that sort-by-codepoint is an order of magnitude more efficient.

It's not a valid choice for anything that actually uses unicode. E.g. if I have functions caféHide() and caféShow() I expect them to be next to each other. I think Java should perhaps have required symbol names to be ASCII, but it doesn't and Java tools should deal with this.

Re: The UTF-8-Everywhere Manifesto

#185

Post by ybungalobill » Tue, May 01, 2012, 9:44 PM UTC

fleitz wrote: post #140

dfox wrote: post #138

Earlier quoted context omitted.

UTF-8 is explicitly designed in such way that unix API dont have to care about it and such that lexicographic ordering of utf-8 encoded byte streams is same as lexicographic ordering of unicode code point vectors (which arguably almost never is what you care about when sorting text strings). Both of these features are result of conscious design and not some random coincidence. As for sorting text strings of any kind…

"When you are sorting bit strings you are sorting lexicographically, but that is usually not what you should be doing." That's my point about it being bullshit, it sounds like a feature you might want, but honestly it's useful to very few people.

Man, the article doesn't try to argue for or against UTF-8 based on lexicographical order. It's in the "facts" section, and it's a correct fact. So it's just as neutral as "Widechar is 2 bytes [...] 4 on others." or "In both [...] characters may take up to 4 bytes".

Re: The UTF-8-Everywhere Manifesto

#186

Post by scoith » Fri, May 04, 2012, 8:42 PM UTC

byuu wrote: post #102

ruediger wrote: post #98

Earlier quoted context omitted.

Can you elaborate on that? Why does Unicode suck for Japanese text?

Not only kanji, but also hiragana and katakana (syllabic alphabets) encode to three bytes per character. Shift-JIS can encode all three to two bytes, as well as half-width katakana to one byte per character. However, if size is such a concern (eg for web transmission), text compression neutralizes the perceived benefit of region-specific encodings. Shift-JIS' continued popularity has much more to do with change avers…

@ruediger There's nothing wrong with Unicode. UTF-8 sucks because it ends up taking more space.

@byuu No it doesn't. Try compressing a SJIS text using gzip. Then convert it to UTF-8 and do the same thing. With a "perfect" compressor, there shouldn't be any difference since the information contents are the same, but unfortunately we don't have a perfect compression algorithm that hits the theoretical lower bound for compression.

Re: The UTF-8-Everywhere Manifesto

#187

Post by ybungalobill » Wed, May 09, 2012, 3:46 PM UTC

khuey wrote: post #93

ori_b wrote: post #56

Earlier quoted context omitted.

http://www.utf8everywhere.org/#faq.cvt.perf If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.

UTF-8 -> UTF-16 conversion was at one point a noticeable fraction of Firefox's startup time. https://bugzilla.mozilla.org/show_bug.cgi?id=506431 Since then we've done things such as fast-path ASCII -> UTF-16 conversion with SSE2 instructions. Converting a few dozen bytes is unlikely to make a significant difference, but often one needs to deal with more than a few dozen bytes.

I believe the authors wrote it clear enough that they don't rule out UTF-16 completely:

> We believe that all other encodings of Unicode (or text, in general) belong to rare edge-cases of optimization and should be avoided by mainstream users.

So if you're writing a browser that must use UTF-16 in its Javascript engine due to dumb standards... it is a reasonable performance optimizations to use UTF-16 for your strings. But how many people write Javascript engines?

Re: The UTF-8-Everywhere Manifesto

#188

Post by ybungalobill » Wed, May 09, 2012, 3:48 PM UTC

twoodfin wrote: post #177

ori_b wrote: post #56

Earlier quoted context omitted.

http://www.utf8everywhere.org/#faq.cvt.perf If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.

It's not the conversion that kills you, it's the memory allocation. If you're trying to interact with a "chatty" UTF-16 API, the difference between using UTF-8 and UTF-16 in your implementation could the difference between passing a pointer (a cycle or two) and allocating/deallocating a buffer (potentially hundreds of cycles) per call . There are obviously ways around this. The API could be rewritten to exchange stri…

I agree in principle. But "chatty" APIs usually work with short strings, in which case you can use stack allocations in those performance critical calls.