Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

101–110 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#101

Post by ajross »

makecheck wrote: post #82
4ad wrote: post #71

Earlier quoted context omitted.

Cat doesn't know and doesn't need to know what kind of input it gets, it's just for concatenatig files. Files themselves can be binary for all that matters (indeed, it was used very often to concatenate tape devices to be used by tar further down in the pipe).

Agreed. The user of "cat" is responsible if they've created a stream that is invalid input to some other program. Unix philosophy is for programs to do (ideally) one simple thing well. If the contents of any one file cannot be used directly, then the idea would be to run some other filter program first (e.g. one that strips bytes from the beginning) prior to using "cat" and piping the results somewhere else. In pract…

...and you think this is an argument in favor of including a BOM?

Re: The UTF-8-Everywhere Manifesto

#102

Post by byuu »

ruediger wrote: post #98
scoith wrote: post #96

That page is misleading when it comes to Japanese text: UTF-8 sucks for Japanese text. UTF-8 and UTF-16 aren't the only two choices within the whole world, which is demonstrated in their choice of encoding Shift-JIS.

Can you elaborate on that? Why does Unicode suck for Japanese text?

Not only kanji, but also hiragana and katakana (syllabic alphabets) encode to three bytes per character. Shift-JIS can encode all three to two bytes, as well as half-width katakana to one byte per character.

However, if size is such a concern (eg for web transmission), text compression neutralizes the perceived benefit of region-specific encodings.

Shift-JIS' continued popularity has much more to do with change aversion than it does technical merit.

Re: The UTF-8-Everywhere Manifesto

#103

Post by njs12345 »

antidoh wrote: post #43

Text is maddening, the modern Tower of Babel. Is there a definitive reference, or small handful of references, to learn all that's worth knowing about text, from ASCII to UTF-∞ and beyond?

Joel Spolsky's 'The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)' is a good start: http://www.joelonsoftware.com/articles/Unicode.html

Like a few other specialised fields (cryptography comes to mind) the key takeaway is to use a library and rely on the work of people who know it better than you do and have handled all the subtleties already :)

Re: The UTF-8-Everywhere Manifesto

#104

Post by sho_hn »

antidoh wrote: post #49
lambda wrote: post #38

Earlier quoted context omitted.

Yeah, it's really sad the number of legacy APIs which have standardized on UTF-16. The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it. I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text…

Some old languages have also made that choice, albeit recently in Python's case: http://www.python.org/dev/peps/pep-3120/

However, when it comes to the internal representation of text, things are quite complex as of Python 3.2: http://www.python.org/dev/peps/pep-0393/

Re: The UTF-8-Everywhere Manifesto

#105

Post by makecheck »

ajross wrote: post #100
makecheck wrote: post #57

Earlier quoted context omitted.

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

What on earth does "processed as such" mean? UTF-8 can be "processed" anywhere ASCII can, that's the whole point. The only point to the BOM is to distinguish it from UTF-16 (or UCS2, which is usually what UTF-16 degenerates into). And UTF-16 is broken garbage and shouldn't be used. Given that, your last sentence is basically an ode to complexity. Taken to the logical conclusion you'd support any rule, no matter how r…

No, UTF-8 cannot be processed in exactly the same way as ASCII! It is highly compatible with ASCII-assuming environments because it can accept the same data (e.g. it remains a stream of bytes that doesn't have to be converted to some "fat" integer) and any UTF-8 text that just happens to contain only ASCII characters will work just fine with an old ASCII program. But complex multi-byte UTF-8 inputs will not work without special treatment.

Historically code pages mapped character values directly to individual bytes: typically redefining the upper half (129+) while leaving the lower numbers the same as ASCII. Programs could display a wide variety of text encodings correctly as long as they knew what the encoding was, and all they had to do was read bytes individually. These encodings could all be handled in a way largely similar to ASCII.

UTF-8 however is a multi-byte encoding, which means you could have (say) 3 bytes that combine to form a symbol. Not only that, but in certain forms multiple symbols could imply display as a single glyph (e.g. an accent followed by a letter). A program that does everything the old ASCII way would choke on multi-byte UTF-8, despite being otherwise-compatible. Consider a program with a fixed-size read buffer; if the last byte is only partway through a multi-byte character, that character will be mishandled unless the program knows how to preserve those bytes and "complete" the character when more bytes arrive.

Re: The UTF-8-Everywhere Manifesto

#106

Post by brigade »

lambda wrote: post #38
pcwalton wrote: post #21

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…

Yeah, it's really sad the number of legacy APIs which have standardized on UTF-16. The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it. I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text…

OS X APIs generally use NSString/CFString, which hide the actual encoding of the string; they can be any encoding at all internally.

Re: The UTF-8-Everywhere Manifesto

#107

Post by luriel »

riffraff wrote: post #75
4ad wrote: post #60

Earlier quoted context omitted.

cat a b c There you go, a BOM in the middle of the file.

if the BOM appears in the middle of the file than it should be considered a zero width non breaking space (so basically ignored) so why is this an issue?

Here is an issue (that has bitten me and took lots of blood and sweat to find out):

cat a b c | grep '^foo'

Re: The UTF-8-Everywhere Manifesto

#108

Post by pjscott »

maaku wrote: post #35
paulsutter wrote: post #15

Earlier quoted context omitted.

The author makes a compelling case for UTF-8 in Asian languages. I'd love to hear any specific counter-arguments.

No he doesn't, he dismisses it out of hand by choosing an example that is uniquely suited to minimize the advantages of UTF-16 for non-Roman scripts. Precious little of an HTML document is actually textual content.

HTML documents are hardly unusual examples. Also, look at the other column in that table, where he stripped out the HTML tags and looked only at the body text: UTF-16 was somewhat smaller, and gzipping them made the difference negligible.

Does UTF-16 really have such a great advantage for non-Roman writing systems? Or is this motivated more by a disliking for Anglocentrism?

Re: The UTF-8-Everywhere Manifesto

#109

Post by brigade »

__david__ wrote: post #51
natch wrote: post #40

Strings (NSString) on Apple platforms are UTF-16. The Apple platforms are not exactly lagging behind in either multilingual, or text processing. I wonder what this team of three people knows that Apple doesn't? Or is it the other way around, that Apple knows something they don't, and when it comes to shipping products that work in the real world, Apple has figured out how to do it?

NSStrings are opaque--you always call accessor functions and never have access to the low level backing store. The reason they are good is that you can't get data into or out of them without specifying an encoding, which leaves the actual encoding of the backing store as an implementation detail. The fact is, I don't even know (or see documented) that the backing is UTF-16--Apple is free to change that at their whim…

It's not documented (presumably) for that very reason.

In fact, the opposite is implied by initWithBytesNoCopy:length:encoding:freeWhenDone: - it should be possible right now to have NSStrings with arbitrary internal representations, even if most other creation methods currently convert to UTF16.

Re: The UTF-8-Everywhere Manifesto

#110

Post by ajross »

makecheck wrote: post #105
ajross wrote: post #100

Earlier quoted context omitted.

What on earth does "processed as such" mean? UTF-8 can be "processed" anywhere ASCII can, that's the whole point. The only point to the BOM is to distinguish it from UTF-16 (or UCS2, which is usually what UTF-16 degenerates into). And UTF-16 is broken garbage and shouldn't be used. Given that, your last sentence is basically an ode to complexity. Taken to the logical conclusion you'd support any rule, no matter how r…

No, UTF-8 cannot be processed in exactly the same way as ASCII! It is highly compatible with ASCII-assuming environments because it can accept the same data (e.g. it remains a stream of bytes that doesn't have to be converted to some "fat" integer) and any UTF-8 text that just happens to contain only ASCII characters will work just fine with an old ASCII program. But complex multi-byte UTF-8 inputs will not work with…

You're thinking from the point of view of a display program that needs to split strings into glyphs. That's a fine application, but very rare. And yes, it's inherently encoding dependent and tends to like wide characters instead of multibyte ones.

But introducing a BOM for the sake of that application is a disaster, because it hurts everything else. You can (literally) feed UTF-8 to parsers written 30 years ago and apply all your existing intuition about string handling in C without worry. Unless you deliberately break it by including a binary, non encoding garbage furball at the front of your "file" (and good luck figuring out what a "file" should mean in a OS metaphor designed around streams).

Post reply on HN