Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

161–170 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#161

Post by jeffdavis »

byuu wrote: post #102
ruediger wrote: post #98

Earlier quoted context omitted.

Can you elaborate on that? Why does Unicode suck for Japanese text?

Not only kanji, but also hiragana and katakana (syllabic alphabets) encode to three bytes per character. Shift-JIS can encode all three to two bytes, as well as half-width katakana to one byte per character. However, if size is such a concern (eg for web transmission), text compression neutralizes the perceived benefit of region-specific encodings. Shift-JIS' continued popularity has much more to do with change avers…

As I said above, I spoke with several Japanese people who said that some valid characters are not representable in Unicode.

Some details can be found here: http://en.wikipedia.org/wiki/Han_unification

Re: The UTF-8-Everywhere Manifesto

#162

Post by pjscott »

mjg59 wrote: post #156
fleitz wrote: post #131

tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else. UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better. There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8. C#…

"it's much less likely to run into issues like String.length returning 32 yet only containing 31 characters" This is exactly the problem with UTF-16. Most APIs that use it will have support for string operations that return the number of codepoints rather than the number of bytes, and as a result people think that it's a solved problem. But in fact you've only solved half the problem, because the number of codepoints…

> (Example: How many characters is "é"? "é"? "é"? Does UTF-16 give you a more useful answer to that question?)

In case anybody was wondering:

1. The first "é" is an "e" followed by a combining acute accent.

2. The second "é" is a single code point for a lowercase-e-with-acute.

3. The third "é" is the same as the second "é", but with a zero-width non-breaking space in front of it.

All of these are, of course, the same letter.

Re: The UTF-8-Everywhere Manifesto

#163

Post by jbert »

makecheck wrote: post #123
jbert wrote: post #84

Earlier quoted context omitted.

The BOM is in-band signalling. It breaks the ASCII-backwards-compatability of UTF8. BOMs are only necessary where the provenance of data is not known. Normally, there is a context provided which will determine the encoding. Basically, yes - text should be tagged (unless 'utf8 everywhere' wins) but imho the tagging should be external to the content.

I agree with the use of context, yes; if you already know your input is UTF-8 (e.g. C strings in a program API or a protocol or whatever) there's no point in adding an extra specifier. If a program requires ASCII compatibility in order to work then by all means make the input files ASCII (no BOM), just make sure the files have no true UTF-8 dependencies in them. Once a program supports UTF-8 "properly" however, the B…

But you have the same problem with text or binary distinctions. On some platforms, no distinction between them needs to be made (quite usefully, adding to tool simplicity and conceptual simplicity).

If you use BOM, a tool which can operate on text or binary must be told which it is operating on. This would have to be done via some external context (e.g. cmdline switch). And that would never go away, even in a utf8 everywhere world.

Basically, in BOM-world: - You either need to tag each fragment of text or you still need to use external context (e.g. which encoding do I get text columns from my database) - You perpetually need to differentiate between binary and text data for all tools which do nothing more complicated than read and write

and in non-BOM-world you: - add to the contextual clues you need anyway something like "any files which you are going to interpret as text on the system should be interpreted as utf8" - when moving data on or off the local the local system, use a network protocol which supports tagging the text payload (e.g. email, http).

The problems arise mostly with file shares (or their equivalent, version control systems) where text files are exchanged without an accompanying protocol. That is where "BOM world" or "UTF8 world" will ultimately have to settle their differences.

BOM-world would like all systems, everywhere, to make a text/binary distinction for ever. UTF8-world would like to say that textual data lacking a context should be interpreted as UTF8. But feel free to use UTF16/UTF32 for specific purposes or systems.

Re: The UTF-8-Everywhere Manifesto

#164

Post by thristian »

jeffdavis wrote: post #160
lmm wrote: post #113

Earlier quoted context omitted.

The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

I spoke with several Japanese people who said that some valid characters are not representable in Unicode. That means that it's not just a technical problem (expensive sort routines or inefficient encodings) -- it's a semantic problem.

The way I've heard it explained, there are some historical alternate versions of some characters (A Latin-alphabet equivalent might be the way we sometimes draw "a" with an extra curl across the top, and sometimes without) that have the exact same semantic meaning, and so they were 'unified' to a single code-point. Unfortunately. some people spell their names exclusively with one variant or the other, and Han unification makes that impossible in Unicode.

Re: The UTF-8-Everywhere Manifesto

#165

Post by pilif »

_3u10 wrote: post #135
pilif wrote: post #86

Really good article. You'll get nothing from me but heartfelt agreement. I especially liked that the article was giving numbers about how inefficient UTF8 would be to store Asian text (not really apparently). Also insightful, but obvious in hindsight: Not even in utf-32 you can index specific character in constant time due to the various digraphs. The one property I really love about UTF8 is that you get a free consi…

"The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string." You don't get this at all using UTF-8. You only get it if you attempt to decode the string which even something like strlen doesn't do. Strlen will happily give you wrong answers about how many characters are in a UTF-8 string all day long and never ever attempt to check th…

>> "The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string."

>You don't get this at all using UTF-8. You only get it if you attempt to decode the string which even something like strlen doesn't do.

I wasn't talking about using strlen (aside of when I was jokingly talking about US-UTF8 where I've seen instances of strlen() being used against UTF-8 strings). I was talking of using library functions designed to handle UTF-8 encoded character data (which strlen() and friends are not).

What I meant with "free consistency check" was that any library function that is designed to deal with UTF-8 data is by default put into a position where it can quite safely determine whether the input data given to it is in fact in UTF-8 or not.

This is not true for any other character encoding I know of (I don't know about the legacy 2-byte Asian encodings at all).

In legacy 8bit character sets, there's nothing you can do to check whether you have been lied to aside of analyzing the content, trying to guess the language and map that to the occurrence of characters in the character set you have been told the string is in (pretty much unfeasible).

With UTF-16 you can at least use some heuristics if you are dealing with common english texts (every second byte would be 0), but you can't be sure - especially not when the text consists of primarily non-ASCII text.

Only with UTF-8 you can take one look at input data and determine with quite a bit of confidence whether the data you have just been handed is in fact in UTF-8 or not (it might still be pure ASCII, but that still qualifies as UTF-8).

If you ever get lied to and somebody tries to feed you ISO-8859-1 claiming it to be UTF-8 (happens all the f'ing time to me), then any library or application designed to deal with UTF-8 can immediately detect this and blow up before you store that data without any way to ever find out what encoding it would have been in.

Re: The UTF-8-Everywhere Manifesto

#166

Post by conradfr »

unconed wrote: post #73
makecheck wrote: post #57

Earlier quoted context omitted.

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

This argument is silly. Why not prefix every UTF-8 string with a BOM then? It's wasteful and unnecessary, because UTF-8's clean structure already makes it trivial to detect, and false positives are all but impossible for real-world text. There's a paper out there that proves this. The UTF-8 BOM was a Microsoft invention. Nobody else uses it, and it breaks tons of things. Two examples off the top of my head: Unix hash…

Not knowing a lot of things about encoding (my bad), BOM wasted me a lot of hours of ajax call debugging in a php app.

Re: The UTF-8-Everywhere Manifesto

#167

Post by danssig »

gavanwoolery wrote: post #25
pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

I agree with you in concept, but the Web is actually not strictly tied to javascript. In the short term (say, next decade), JS is probably not going anywhere, but in the longer term I hope someone creates a more well-thought-out replacement. (And for the record, I kind of like javascript, just a few things I would change with it.)

Yes, like a proper VM that we can compile our preferred language to. It is ridiculous that the "language of the web" is something that is forced upon us instead of chosen. The number of languages that compile to JS illustrate the point nicely.

Re: The UTF-8-Everywhere Manifesto

#168

Post by tripzilch »

makecheck wrote: post #121
ajross wrote: post #101

Earlier quoted context omitted.

...and you think this is an argument in favor of including a BOM?

I prefer to solve the problem in the right place. Raw concatenation of bytes without encoding-awareness introduces the possibility that the bytes will combine in unexpected ways. The presence of something as obvious as a BOM makes it harder to make this mistake, at least during the transition phase to "UTF-8 everywhere". What you really want in this situation is something that forces you to see the potential bug and…

So, first you argue that UTF-8 without BOM might split glyphs at incorrect places, and then it's pointed out that UTF-8 with BOM concatenates files incorrectly, and somehow this is also an argument for using BOM?

I'm sorry but it's obvious both approaches have rather symmetrical opportunities to produce bugs. So in that case the best choice seems to be the simplest one. Which is the one without BOM.

Re: The UTF-8-Everywhere Manifesto

#169

Post by jeltz »

muyuu wrote: post #126
lambda wrote: post #34

Earlier quoted context omitted.

Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything o…

The author compares UTF-8 to UTF-16, while there are a myriad better encodings than both for different Asian languages. For instance: EUC-JP in Japan, BIG5 in Taiwan, GB in China, etc. Different EUC encodings are variable-length and are a lot more efficient since they put common subsets in lower parts of the table for each language, close to each other so they also compress better, while allowing tricks for text matc…

> Note that gzipped or otherwise compressed text makes differences even worse at least in the case of Japanese - where UTF-8 text gets de-aligned all over the place to odd byte sizes and compresses worse.

That was not what this article measured. UTF-8 and UTF-16 compressed to virtually the same size.

Re: The UTF-8-Everywhere Manifesto

#170

Post by jeltz »

lmm wrote: post #113
haberman wrote: post #54

Totally agree re: UTF-8 vs other Unicode encodings. But are there still still hold-outs who don't like Unicode? Last I heard some CJK users were unhappy about Han Unification: http://en.wikipedia.org/wiki/Han_unification

The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

Sort by unicode codepoint does not work well in most western languages either. It is almost only English where it works good enough to be usable among the languages with a latin script.

For example the sort order is broken for all Nordic languages.

Post reply on HN