Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

171–180 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#171

Post by cygx »

Personally, I prefer UTF-8 as well. However, I think this whole debate about choice of encoding gets blown out of proportion.

Consider the following diagram:

                               [user-perceived characters]  [grapheme clusters]       |
                       ^                    ^                |
                       |                    |                |
                       v                    v                |
      [bytes]  [codepoints]           [glyphs]
Choice of encoding only affects the conversion from bytes to codepoints, which is pretty straight-forward: The subtleties lie elsewhere...

Re: The UTF-8-Everywhere Manifesto

#172

Post by ori_b »

kijin wrote: post #127
ori_b wrote: post #41

Earlier quoted context omitted.

utf-8 is a single byte encoding. Reversing the order of a sequence that's one byte long just gives back that one byte.

No it isn't. Any letter with an accent will take up two bytes. Most non-Latin characters take up three bytes, sometimes even four.

Poorly phrased. It can take multiple bytes to fully define one codepoint, but the encoding is defined in terms of a stream of single bytes. In other words, each unit is one byte, hence flipping each unit gives back the same unit.

This is not the case for UTF-16 and UTF-32.

Re: The UTF-8-Everywhere Manifesto

#173

Post by ori_b »

makecheck wrote: post #57
luriel wrote: post #31

Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

What would you do differently if the mark was there versus if it wasn't?

ASCII is a perfect subset of UTF-8, so any operations you would do on UTF-8 are also operations you would do on ASCII. The behavior of your program wouldn't change. The BOM is a no-op when it comes to how your program handles things.

However, it has the potential of confusing older programs. It makes things that should be simple (like 'cat') need to be encoding aware, and modal. It means that streaming multiple files one after the other breaks.

So, in summary, the BOM doesn't buy you anything, or give you any important information about a file. It does make things harder.

It should die.

Re: The UTF-8-Everywhere Manifesto

#174

Post by mokus »

alecco wrote: post #152
alecco wrote: post #12

ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low. Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

The new HN: disagree = downvote

It's not really new.

Whether you agree with the sentiment or not, pg's oldish comment on the issue at least establishes the existence of the behavior several years ago, and has been taken by many to be the final word on the acceptability of the practice: http://news.ycombinator.com/item?id=117171

(edit: spelling)

Re: The UTF-8-Everywhere Manifesto

#175

Post by adavies42 »

jeffdavis wrote: post #160
lmm wrote: post #113

Earlier quoted context omitted.

The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

I spoke with several Japanese people who said that some valid characters are not representable in Unicode. That means that it's not just a technical problem (expensive sort routines or inefficient encodings) -- it's a semantic problem.

isn't the real problem that you can't guarantee correct rendering of ideograph text without specifying fonts? there are japanese kanji that are drawn differently from the chinese hanzi they're descended from, but they're the same from a unicode perspective.

imagine if roman, greek, cyrillic, hebrew (aramaic), and ethiopian (ge'ez) were all assigned to the same group of code points and distinguishable only by font--they're all just variants of phoenician, after all....

Re: The UTF-8-Everywhere Manifesto

#176

Post by mikelward »

_3u10 wrote: post #135
pilif wrote: post #86

Really good article. You'll get nothing from me but heartfelt agreement. I especially liked that the article was giving numbers about how inefficient UTF8 would be to store Asian text (not really apparently). Also insightful, but obvious in hindsight: Not even in utf-32 you can index specific character in constant time due to the various digraphs. The one property I really love about UTF8 is that you get a free consi…

"The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string." You don't get this at all using UTF-8. You only get it if you attempt to decode the string which even something like strlen doesn't do. Strlen will happily give you wrong answers about how many characters are in a UTF-8 string all day long and never ever attempt to check th…

"One was an extended ASCII string that happend to be valid UTF-8 sans BOM :)"

Do you mean you pasted in a string of bytes that was valid UTF-8 into an app expecting UTF-8, and it didn't decide to convert it into ISO 8859-something based on some heuristic?

Sounds like correct behavior to me.

Re: The UTF-8-Everywhere Manifesto

#177

Post by twoodfin »

ori_b wrote: post #56
pcwalton wrote: post #27

Earlier quoted context omitted.

Well, we're talking about DOM manipulation performance here. Pages that use DOM manipulation heavily will see a potentially-unacceptable performance loss if text always has to be converted to UTF-8. Is fast DOM manipulation important? Given that the only way for the sole scripting language on the Web to display anything or interact with the user is through DOM manipulation, I think it's worth optimizing every cycle..…

http://www.utf8everywhere.org/#faq.cvt.perf If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.

It's not the conversion that kills you, it's the memory allocation. If you're trying to interact with a "chatty" UTF-16 API, the difference between using UTF-8 and UTF-16 in your implementation could the difference between passing a pointer (a cycle or two) and allocating/deallocating a buffer (potentially hundreds of cycles) per call.

There are obviously ways around this. The API could be rewritten to exchange strings less frequently or use static strings that could be replaced with handles. You can try to be clever about your buffer allocation and share one amongst all calls (but watch out for threading issues!) You could write your own allocator. But all this plumbing just increases complexity and the risk of bugs, along with adding its own performance cost.

I'm not arguing against UTF-8 as the preferred encoding for many future applications, but the "minimal overhead" example given in the manifesto isn't particularly convincing.

Re: The UTF-8-Everywhere Manifesto

#178

Post by alecco »

lambda wrote: post #34
alecco wrote: post #12

ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low. Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything o…

> What do you mean by a "proper way"? If size is what you care about, just compress your data. Compression will do a lot better for a much wider range of data than some clever encoding will.

That's precisely what I meant. Simple variable-length compression.

Re: The UTF-8-Everywhere Manifesto

#179

Post by jeltz »

fleitz wrote: post #148
jsprinkles wrote: post #146

Earlier quoted context omitted.

> You only get it if you attempt to decode the string which even something like strlen doesn't do. Because strlen() is a count of chars in a null-terminated char[], not a decoder. Ever. It's character set agnostic. > Strlen will happily give you wrong answers about how many characters are in a UTF-8 string all day long and never ever attempt to check the validity of the string. Because, again, strlen() counts chars i…

A name like strlen suggests that it's designed to take the length of a string, if it was called count_null_ter_char_array then I'd tend to believe you. It's not character set agonistic, it's monotheistic at the shrine of ASCII, it's all over the coding style. Null is valid UTF-8, it just doesn't work with C 'strings'. I can get null out of a UTF-8 encoder with no problem. My point is that UTF-8 is nowhere near the pa…

I do not understand how UTF-16 could be better for this reason. wcslen works exactly like strlen but on wide chars instead of chars.

Re: The UTF-8-Everywhere Manifesto

#180

Post by alecco »

mokus wrote: post #174
alecco wrote: post #152

Earlier quoted context omitted.

The new HN: disagree = downvote

It's not really new. Whether you agree with the sentiment or not, pg's oldish comment on the issue at least establishes the existence of the behavior several years ago, and has been taken by many to be the final word on the acceptability of the practice: http://news.ycombinator.com/item?id=117171 (edit: spelling)

Paired with increasing groupthink, HN discussions are getting boring.
Post reply on HN