Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The UTF-8-Everywhere Manifesto

utf8everywhere.org

Post reply on HN
Print view View original

71–80 of 188 posts

Re: The UTF-8-Everywhere Manifesto

#71

Post by 4ad »

adobriyan wrote: post #66
4ad wrote: post #60

Earlier quoted context omitted.

cat a b c There you go, a BOM in the middle of the file.

Unicode-compatible cat(1) would strip BOM from all but the first file. Applying non-Unicode compatible utility to Unicode files of course doesn't work.

Cat doesn't know and doesn't need to know what kind of input it gets, it's just for concatenatig files. Files themselves can be binary for all that matters (indeed, it was used very often to concatenate tape devices to be used by tar further down in the pipe).

Re: The UTF-8-Everywhere Manifesto

#72

Post by mkup »

I use UTF-8 for transmitted data and disk I/O, and I use UCS-4 (wchar_t on Linux/FreeBSD) for internal representation of strings in my software.

I generally agree with this article, but I disagree with it on the point that UTF-8 is the only appropriate encoding for strings stored in memory, and also I disagree on the point wchar_t should be removed from C++ standard or made sizeof 1, as in Android NDK.

Let me explain why.

In UTF-8 single Unicode character may be encoded in multiple ways. For example NUL (U+0000) can be encoded as 00 or as C0 80. The second encoding is illegal because it's longer than necessary and forbidden by standard, but naive parser may extract NUL out of it. If UTF-8 input was not properly sanitized, or there is a bug in charset converter, this may result in exploit like SQL injection or arbitrary filesystem access or something like that: malicious party can encode not only NUL, but ", /, \ etc this way.

Also UTF-8 string can't be cut at arbitrary position. Byte groups (UTF-8 runes) must be processed as a whole, so appear either on left side or on the right side of cut.

Reversing of UTF-8 string is tricky, especially when illegal character sequences are present in input string and corresponding code points (U+FFFD) must be preserved in output string.

I think UTF-8 for network transmitted data and disk I/O is inevitable, but our software should keep all in-memory strings in UCS-4 only, and take adequate security precautions in all places where conversion between UTF-8 and UCS-4 happens.

And sizeof(wchar_t)==4 in GCC ABI is not a design defect, wchar_t exists for a good reason. I admit that sizeof(wchar_t)==2 on Windows is utterly broken.

Re: The UTF-8-Everywhere Manifesto

#73

Post by unconed »

makecheck wrote: post #57
luriel wrote: post #31

Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

This argument is silly. Why not prefix every UTF-8 string with a BOM then? It's wasteful and unnecessary, because UTF-8's clean structure already makes it trivial to detect, and false positives are all but impossible for real-world text. There's a paper out there that proves this.

The UTF-8 BOM was a Microsoft invention. Nobody else uses it, and it breaks tons of things. Two examples off the top of my head: Unix hashbang scripts (i.e. #!/bin/bash), and PHP scripts (the BOM will trigger HTTP header finalization before any code is run).

Re: The UTF-8-Everywhere Manifesto

#74

Post by bobbydavid »

erichocean wrote: post #65
lubutu wrote: post #61

Earlier quoted context omitted.

No. NUL is backwards-compatible with ASCII, and is used everywhere. Choosing some arbitrary invalid UTF-8 byte for use as a terminator would be a terrible decision. If you want to handle NUL, simply use length-annotated slices instead of C-style NUL-terminated strings. Anything else is completely wrong.

Did you even read my comment? NULL is a valid UTF-8 character. If specific languages and their standard libraries choose to treat it as a string terminator (C, I'm looking at you), well, then fine. But it's still valid Unicode. If you claim to support Unicode strings, but don't support the NULL character in those strings, you don't support Unicode strings in their entirety. Going further, there's nothing in the ASCII…

I think what you are trying to say is:

"because UTF-8 has invalid character sequences, we could potentially use one of them to represent end-of-string, which would allow us the flexibility of a null-terminated string (not keeping track of the length) without the restriction of no-nulls-allowed."

You're right! Great. But you are not revealing a "strange thing" about Unicode. You are instead making a general comment about null-terminated strings. So why use such inflammatory and misleading language like "If you claim to support Unicode, you have to support NULL characters"?

Update: I don't object to your idea at all, it's a neat trick! It's just that the way it's phrased, it sounds like Unicode's design contributed to this NULL-terminal problem, when in fact even NULL-terminated ASCII strings cannot 'handle' a null character in this sense.

To augment your idea, though, how about you use '0xFF 0x00' as a terminator? This way, backward-compatibility is preserved in all cases except UTF-8 => ASCII with NULLs, and in this case the string will be truncated rather than a buffer overflow (i.e. "fail closed").

Re: The UTF-8-Everywhere Manifesto

#75

Post by riffraff »

4ad wrote: post #60
makecheck wrote: post #57

Earlier quoted context omitted.

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

cat a b c There you go, a BOM in the middle of the file.

if the BOM appears in the middle of the file than it should be considered a zero width non breaking space (so basically ignored) so why is this an issue?

Re: The UTF-8-Everywhere Manifesto

#76

Post by pornel »

CJefferson wrote: post #33

I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness? In particular, what should I watch out for to make an ASCII parser UTF-8 clean?

UTF-8 is easy to figure out. However, it's only a doorway to Unicode, and Unicode is not simple.

If you do anything more than chopping codepoints and passing them around — use a library. There's a lot of technical complexity due to quirks of Unicode and inherent complexity of world's diverse writing systems.

Avoid using concept of a "character" as much as you can, as it's fuzzy, e.g. there are combining characters and ligatures (codepoint != character).

Be aware that string comparison cannot be done just by comparing codepoints, and there are different levels of "sameness" of Unicode strings coming from different normalisations, e.g. NFC and NFKD.

Case-insensitive comparison cannot be done by lowercasing a string: http://www.moserware.com/2008/02/does-your-code-pass-turkey-...

Unfortunately I don't know much about RTL text, and there's lots of traps there too (e.g. there are control characters for controlling text direction).

Re: The UTF-8-Everywhere Manifesto

#77

Post by erichocean »

erichocean wrote: post #65
lubutu wrote: post #61

Earlier quoted context omitted.

No. NUL is backwards-compatible with ASCII, and is used everywhere. Choosing some arbitrary invalid UTF-8 byte for use as a terminator would be a terrible decision. If you want to handle NUL, simply use length-annotated slices instead of C-style NUL-terminated strings. Anything else is completely wrong.

Did you even read my comment? NULL is a valid UTF-8 character. If specific languages and their standard libraries choose to treat it as a string terminator (C, I'm looking at you), well, then fine. But it's still valid Unicode. If you claim to support Unicode strings, but don't support the NULL character in those strings, you don't support Unicode strings in their entirety. Going further, there's nothing in the ASCII…

@bobbydavid Thanks for re-stating what I'm saying.

Re: "So why use such inflammatory and misleading language like "If you claim to support Unicode, you have to support NULL characters"?"

I'm not trying to be "inflammatory" or "misleading". IMO (note: opinion), if a given API claims to support Unicode strings, but instead disallows certain Unicode characters from appearing in those strings, then, IMO, that API only partially supports Unicode strings.

Others could have different opinions (evidently, you do).

I don't know if handling all Unicode characters should be a requirement for an implementation to call itself "Unicode-compliant", but barring another standard, that seams reasonable to me.

A better option IMO would be to have never included NULL in Unicode in the first place, since it is so widely used in system software as an end-of-string terminator. But that ship has sailed...

Re: The UTF-8-Everywhere Manifesto

#78

Post by makecheck »

unconed wrote: post #73
makecheck wrote: post #57

Earlier quoted context omitted.

The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…

This argument is silly. Why not prefix every UTF-8 string with a BOM then? It's wasteful and unnecessary, because UTF-8's clean structure already makes it trivial to detect, and false positives are all but impossible for real-world text. There's a paper out there that proves this. The UTF-8 BOM was a Microsoft invention. Nobody else uses it, and it breaks tons of things. Two examples off the top of my head: Unix hash…

You wouldn't prefix every string with it because presumably your API or program's state has already determined the string's encoding. I am not suggesting that every fragment of text has to be explicit (I agree that would be ridiculous). I am only stating facts: there is nothing incorrect about having the mark, a conformant reader must be able to handle the mark, and the mark has some value as a short-cut for avoiding elaborate decoding tricks.

Re: The UTF-8-Everywhere Manifesto

#79

Post by ori_b »

pcwalton wrote: post #70
ori_b wrote: post #56

Earlier quoted context omitted.

http://www.utf8everywhere.org/#faq.cvt.perf If the function you're calling with UTF8 is non-trivial, converting a few dozen bytes is unlikely to make a significant difference. Benchmark it, of course, but don't be surprised if you don't need to care. Modifying the DOM is probably going to be non-trivial.

See my comment above; I can construct cases that make this sort of conversion have unacceptable overhead. Would it matter in a real-world setting? I can't say for sure, because nobody I know of has tried making a production-quality UTF-8 web layout engine. But, in my mind, none of the benefits of UTF-8 (memory usage being the main one in a browser [1]) outweigh the performance risks of doing conversion. And the risk…

Is string storage really that big a portion of the browser's memory usage? I find it hard to believe that my browser is currently storing nearly a gigabyte of text right now I'm not saying that there wouldn't be performance overhead, but would it be significant? I would be surprised if it made a big difference.

For legacy reasons, there probably isn't a point in changing it. But I'd be surprised if performance reasons turned out to be gating.

Post reply on HN