Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

121–130 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#121

Post by makecheck »

ajross wrote: post #101
makecheck wrote: post #82

Earlier quoted context omitted.

Agreed. The user of "cat" is responsible if they've created a stream that is invalid input to some other program. Unix philosophy is for programs to do (ideally) one simple thing well. If the contents of any one file cannot be used directly, then the idea would be to run some other filter program first (e.g. one that strips bytes from the beginning) prior to using "cat" and piping the results somewhere else. In pract…

...and you think this is an argument in favor of including a BOM?

I prefer to solve the problem in the right place.

Raw concatenation of bytes without encoding-awareness introduces the possibility that the bytes will combine in unexpected ways. The presence of something as obvious as a BOM makes it harder to make this mistake, at least during the transition phase to "UTF-8 everywhere".

What you really want in this situation is something that forces you to see the potential bug and introduce the correct translation and/or text-concatenation tools to fix it.

And yes, at some point in the future enough tools will be truly aware of UTF-8 that the BOM will not have a reason to exist. But right now it has some value.

Re: The UTF-8-Everywhere Manifesto

#122

Post by makecheck »

ajross wrote: post #110
makecheck wrote: post #105

Earlier quoted context omitted.

No, UTF-8 cannot be processed in exactly the same way as ASCII! It is highly compatible with ASCII-assuming environments because it can accept the same data (e.g. it remains a stream of bytes that doesn't have to be converted to some "fat" integer) and any UTF-8 text that just happens to contain only ASCII characters will work just fine with an old ASCII program. But complex multi-byte UTF-8 inputs will not work with…

You're thinking from the point of view of a display program that needs to split strings into glyphs. That's a fine application, but very rare. And yes, it's inherently encoding dependent and tends to like wide characters instead of multibyte ones. But introducing a BOM for the sake of that application is a disaster, because it hurts everything else. You can (literally) feed UTF-8 to parsers written 30 years ago and a…

If a file really is pure ASCII, leave it that way. I am not suggesting to do otherwise. If a 30-year-old program only deals with ASCII then make sure your input looks like ASCII.

But if your input could contain complex UTF-8 (e.g. it's multi-language or whatever), you're not doing any favors by hiding this fact. The BOM is a quick way to know exactly what the file is, and it shows you that your program won't work with that input. So you translate the input or you fix the program.

At some point in the future the majority of programs will handle even complex UTF-8 properly, and then the BOM will be pointless because virtually all inputs will be UTF-8.

Re: The UTF-8-Everywhere Manifesto

#123

Post by makecheck »

jbert wrote: post #84
makecheck wrote: post #57

Earlier quoted context omitted.

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

The BOM is in-band signalling. It breaks the ASCII-backwards-compatability of UTF8. BOMs are only necessary where the provenance of data is not known. Normally, there is a context provided which will determine the encoding. Basically, yes - text should be tagged (unless 'utf8 everywhere' wins) but imho the tagging should be external to the content.

I agree with the use of context, yes; if you already know your input is UTF-8 (e.g. C strings in a program API or a protocol or whatever) there's no point in adding an extra specifier.

If a program requires ASCII compatibility in order to work then by all means make the input files ASCII (no BOM), just make sure the files have no true UTF-8 dependencies in them.

Once a program supports UTF-8 "properly" however, the BOM is useful as a signal that the input is somewhat complicated.

At some point in the future when UTF-8 really is everywhere and programs may no longer even try to sniff encodings, etc. then yes, the BOM has no real reason to exist.

Re: The UTF-8-Everywhere Manifesto

#124

Post by yuhong »

pilif wrote: post #86

Really good article. You'll get nothing from me but heartfelt agreement. I especially liked that the article was giving numbers about how inefficient UTF8 would be to store Asian text (not really apparently). Also insightful, but obvious in hindsight: Not even in utf-32 you can index specific character in constant time due to the various digraphs. The one property I really love about UTF8 is that you get a free consi…

Yea, reminds me of DBCS. UTF-8 however don't use bytes below 0x80 as anything other than an ASCII character, unlike some DBCS encodings such as Shift-JIS.

Re: The UTF-8-Everywhere Manifesto

#125

Post by makecheck »

unconed wrote: post #120
makecheck wrote: post #78

Earlier quoted context omitted.

You wouldn't prefix every string with it because presumably your API or program's state has already determined the string's encoding. I am not suggesting that every fragment of text has to be explicit (I agree that would be ridiculous). I am only stating facts: there is nothing incorrect about having the mark, a conformant reader must be able to handle the mark, and the mark has some value as a short-cut for avoiding…

The thing is, the BOM is metadata, it doesn't belong in content. It violates the contract of .txt files, which is: the entire file is a single string of content. Recognizing it at the edges of your program and stripping it out is not the end of the world, but it's annoying and no other (8-bit) encoding works that way. In fact, I find it hard to believe UTF-8 BOMs in MS programs were anything more than a programmer er…

There are already plenty of cases that valid UTF-8 readers have to deal with (unused ranges of code points, invalid byte combinations, etc.). Ignoring a BOM is trivial by comparison. A UTF-8 reader honestly doesn't care about the "stringness" of a .txt file because of all the other crap that can be in a byte stream.

Older programs do care, but as I've said elsewhere in the thread an ASCII file can remain ASCII (no BOM). There's no reason to BOM-ify an old ASCII file if it really is ASCII and only ASCII-expecting programs will ever use it.

Over time these old programs will either be upgraded or go away and it will finally be safe to say that inputs must be UTF-8. At that time, the BOM has no reason to exist.

Re: The UTF-8-Everywhere Manifesto

#126

Post by muyuu »

lambda wrote: post #34
alecco wrote: post #12

ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low. Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything o…

The author compares UTF-8 to UTF-16, while there are a myriad better encodings than both for different Asian languages.

For instance: EUC-JP in Japan, BIG5 in Taiwan, GB in China, etc. Different EUC encodings are variable-length and are a lot more efficient since they put common subsets in lower parts of the table for each language, close to each other so they also compress better, while allowing tricks for text matching and searches (not really necessary for web sites, but it's nice to use the same encoding throughout applications sometimes). Russian and Greek are also basically multiplied by two in size.

There are a lot of other considerations.

If you think a 30%+ saving in size (and latency) is not a big deal, then you're a lot more likely to lose to local competitors. Note that gzipped or otherwise compressed text makes differences even worse at least in the case of Japanese - where UTF-8 text gets de-aligned all over the place to odd byte sizes and compresses worse. Add to that the fact that Asians browse the net A LOT from the phone and have done so for much longer than westerners and in a bigger percentage, and you have your problem exacerbated even further.

There is a lot more to consider and like it or not it's not as simple as "UTF-8 for everything and everybody, ever!"

Re: The UTF-8-Everywhere Manifesto

#127

Post by kijin »

ori_b wrote: post #41
sopooneo wrote: post #29

Earlier quoted context omitted.

Thank you. That is very good to learn and I looked over the wikipedia article. But as far as byte order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?

utf-8 is a single byte encoding. Reversing the order of a sequence that's one byte long just gives back that one byte.

No it isn't. Any letter with an accent will take up two bytes. Most non-Latin characters take up three bytes, sometimes even four.

Re: The UTF-8-Everywhere Manifesto

#128

Post by kijin »

muyuu wrote: post #126
lambda wrote: post #34

Earlier quoted context omitted.

Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything o…

The author compares UTF-8 to UTF-16, while there are a myriad better encodings than both for different Asian languages. For instance: EUC-JP in Japan, BIG5 in Taiwan, GB in China, etc. Different EUC encodings are variable-length and are a lot more efficient since they put common subsets in lower parts of the table for each language, close to each other so they also compress better, while allowing tricks for text matc…

Have you ever worked with a system that needs to deal with more than one language at the same time? What if your users want to mix Japanese with Russian in the same sentence? Or Japanese and simplified Chinese? (Yes, people do that.)

In the global Internet, UTF-16 and UTF-8 are the only games in town.

Re: The UTF-8-Everywhere Manifesto

#129

Post by kragen »

adobriyan wrote: post #66
4ad wrote: post #60

Earlier quoted context omitted.

cat a b c There you go, a BOM in the middle of the file.

Unicode-compatible cat(1) would strip BOM from all but the first file. Applying non-Unicode compatible utility to Unicode files of course doesn't work.

Applying non-Unicode-compatible utilities to UTF-8 files works fine; that's why UTF-8 was invented. Adding a byte-order mark to a UTF-8 file is what causes non-Unicode-compatible utilities to stop working.

Re: The UTF-8-Everywhere Manifesto

#130

Post by gwillen »

makecheck wrote: post #57
luriel wrote: post #31

Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

The BOM is not legal UTF-8. Do not put a BOM in your UTF-8 documents. (Accept a BOM in other people's UTF-8 documents, because Postel's Law, and because other people are dumb. But don't perpetuate it...)
Post reply on HN