Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

151–160 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#151

Post by khuey »

byuu wrote: post #102
ruediger wrote: post #98

Earlier quoted context omitted.

Can you elaborate on that? Why does Unicode suck for Japanese text?

Not only kanji, but also hiragana and katakana (syllabic alphabets) encode to three bytes per character. Shift-JIS can encode all three to two bytes, as well as half-width katakana to one byte per character. However, if size is such a concern (eg for web transmission), text compression neutralizes the perceived benefit of region-specific encodings. Shift-JIS' continued popularity has much more to do with change avers…

On the web ASCII (think HTML tags, CSS stylesheets, etc) typically is a large fraction of CJK pages, so the relative inefficiency of UTF-8 for encoding is less important.

Re: The UTF-8-Everywhere Manifesto

#152

Post by alecco »

alecco wrote: post #12

ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low. Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

The new HN: disagree = downvote

Re: The UTF-8-Everywhere Manifesto

#153

Post by derleth »

muyuu wrote: post #149
derleth wrote: post #144

Earlier quoted context omitted.

> It's not like you have to use the same encoding all the time. Then you are going to feed someone garbage. Why feed people garbage?

?? Not if you know what you're doing. Not any more than using utf8 exclusively all the time and for all purposes.

> Not if you know what you're doing.

This is nice in theory. In practice, people make mistakes. Make it easy on yourself.

> Not any more than using utf8 exclusively all the time and for all purposes.

Maybe I was unclear: Feeding me Chinese text in UTF-8 is not garbage. Feeding me anything in one of the GB encodings is garbage.

Garbage, to me, is text in an encoding I can't handle. If you only use UTF-8, that cannot possibly happen.

Re: The UTF-8-Everywhere Manifesto

#154

Post by colanderman »

erichocean wrote: post #59
colanderman wrote: post #58

Earlier quoted context omitted.

-1 is not a valid Unicode code point. "All 1s" is not adequately defined without saying how many 1s – and Unicode does not specify a maximum bit width. Even if you said "the maximum Unicode code point", that is not all 1s – it is 0x10FFFF.

That's the entire point of choosing -1 as an "end of sequence" marker for a UTF-8 string when the length is not known up front. A byte containing all 1s is not valid in any Unicode encoding, so if one appears, you'd know you had hit the end of the string.

OK, I thought you meant a code point containing all 1s. Thanks for clearing that up.

Re: The UTF-8-Everywhere Manifesto

#155

Post by jsprinkles »

fleitz wrote: post #131

tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else. UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better. There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8. C#…

> UNIX isn't UTF-8

These are the only three words that are correct in your comment. You're spreading a lot of FUD about UTF-8, and in many cases, you are completely incorrect about many things technical.

I hate to make this personal, but, you really need to investigate what you're talking about before jumping on HN and talking shit about UTF-8. UTF-8 has some criticisms, but none of what you have written so far is even remotely valid. I hope people realize this instead of get scared away which is, interestingly, what the point of the manifesto linked above is all about.

Re: The UTF-8-Everywhere Manifesto

#156

Post by mjg59 »

fleitz wrote: post #131

tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else. UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better. There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8. C#…

"it's much less likely to run into issues like String.length returning 32 yet only containing 31 characters"

This is exactly the problem with UTF-16. Most APIs that use it will have support for string operations that return the number of codepoints rather than the number of bytes, and as a result people think that it's a solved problem. But in fact you've only solved half the problem, because the number of codepoints is almost certainly not what you want - you want the number of characters, and the only way your library functions can know that is to know which Unicode codepoints are combining characters. And that set potentially gets larger with every new Unicode release.

In other words, if you're relying on languages that have native UTF-16 support to tell you the number of printable characters, your application is inevitably going to be broken the first time someone uses a newly-defined combining character. UTF-16 buys you absolutely nothing useful in this respect.

(Example: How many characters is "é"? "é"? "é"? Does UTF-16 give you a more useful answer to that question?)

Re: The UTF-8-Everywhere Manifesto

#157

Post by vorg »

fleitz wrote: post #131

tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else. UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better. There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8. C#…

> "UTF-8 and UTF-32 result the same order when sorted lexicographically. UTF-16 does not." This is complete and utter bullshit, to sort a string lexicographically you need to decode it, if you've decoded the string into UNICODE then they sort the exact same way.

One original purpose of Unicode, still mentioned in the published for v6.0, is to assist the 100-odd other encodings, not replace them. Each other encoding only needs a conversion process to and from Unicode, i.e. 200 conversion processes overall, instead of into every other encoding, i.e. 9900 processes overall. For sorting, Unicode is order-invariant. The text should be converted into a relevant country-specific encoding, sorted within that encoding, then converted back to Unicode.

Re: The UTF-8-Everywhere Manifesto

#158

Post by yaix »

makecheck wrote: post #57
luriel wrote: post #31

Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

Just assume that it is UTF-8, no sniffing needed. If it's not, then convert it to UTF-8.

Re: The UTF-8-Everywhere Manifesto

#159

Post by kenmazy »

derleth wrote: post #132
sanxiyn wrote: post #117

Earlier quoted context omitted.

I am a Korean user (K in CJK), and no one, I repeat, no one, care about Han unification here. I heard that it is different in China and Japan though.

Probably because modern Korean text is Hangul, which is not really derived from the Han characters Chinese and Japanese have in common. http://en.wikipedia.org/wiki/Hangul http://en.wikipedia.org/wiki/Chinese_characters

Hanja is widely used in modern Korea.

http://en.wikipedia.org/wiki/Hanja

Re: The UTF-8-Everywhere Manifesto

#160

Post by jeffdavis »

lmm wrote: post #113
haberman wrote: post #54

Totally agree re: UTF-8 vs other Unicode encodings. But are there still still hold-outs who don't like Unicode? Last I heard some CJK users were unhappy about Han Unification: http://en.wikipedia.org/wiki/Han_unification

The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

I spoke with several Japanese people who said that some valid characters are not representable in Unicode.

That means that it's not just a technical problem (expensive sort routines or inefficient encodings) -- it's a semantic problem.

Post reply on HN