81–90 of 188 posts
Earlier quoted context omitted.
Unicode-compatible cat(1) would strip BOM from all but the first file. Applying non-Unicode compatible utility to Unicode files of course doesn't work.
Cat doesn't know and doesn't need to know what kind of input it gets, it's just for concatenatig files. Files themselves can be binary for all that matters (indeed, it was used very often to concatenate tape devices to be used by tar further down in the pipe).
In practice though some programs and protocols have text/binary distinction (e.g. FTP). It's not unreasonable to have a mode that hints when bytes are to be used as text. This is frequently done to handle different new-line styles for instance.
Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks. For new APIs in which legacy interoperabil…
I agree with you in concept, but the Web is actually not strictly tied to javascript. In the short term (say, next decade), JS is probably not going anywhere, but in the longer term I hope someone creates a more well-thought-out replacement. (And for the record, I kind of like javascript, just a few things I would change with it.)
Yes! I have been meaning to write something like this for years. There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files. Edit: Here is the interesting story of how Ken Thompson invented UTF-8: http://doc.cat-v.org/bell_labs/utf-8_history
The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly. Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 correctly then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it…
BOMs are only necessary where the provenance of data is not known. Normally, there is a context provided which will determine the encoding.
Basically, yes - text should be tagged (unless 'utf8 everywhere' wins) but imho the tagging should be external to the content.
Earlier quoted context omitted.
The author makes a compelling case for UTF-8 in Asian languages. I'd love to hear any specific counter-arguments.
No he doesn't, he dismisses it out of hand by choosing an example that is uniquely suited to minimize the advantages of UTF-16 for non-Roman scripts. Precious little of an HTML document is actually textual content.
Most of his points are not related to encoding size but to simplicity and standardization. I find those reasons to be very compelling.
I'm asking for clear counterarguments because I concede that my ASCII background could predispose me to UTF-8. Please be a little clearer, I really do want to know.
Also insightful, but obvious in hindsight: Not even in utf-32 you can index specific character in constant time due to the various digraphs.
The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string.
This is a really good help for detecting encoding errors very early (still to this day, applications are known to lie about the encoding of their output).
And of course, there's no endianness issue, removing the need for a BOM which makes it possible for tools that operate at byte levels to still do the right job.
If only it had better support outside of Unix.
For example, try opening a UTF8 encoded CSV file (using characters outside of ASCII of course) in Mac Excel (latest versions. Up until that, it didn't know UTF8 at all) for a WTF experience somewhere between comical and painful.
If there is one thing I could criticize about UTF8 then that would be its similarity to ASCII (which is also its greatest strength) causing many applications and APIs to boldly declare UTF8 compatibility when all they really can do is ASCII compatibility and emitting a mess (or blowing up) once they have to deal with code points outside that range.
I'm jokingly calling this US-UTF8 when I encounter it (all too often unfortunately), but maybe the proliferation of "cool" characters like what we recently got with Emoji is likely going to help with this over time.
I use UTF-8 for transmitted data and disk I/O, and I use UCS-4 (wchar_t on Linux/FreeBSD) for internal representation of strings in my software. I generally agree with this article, but I disagree with it on the point that UTF-8 is the only appropriate encoding for strings stored in memory, and also I disagree on the point wchar_t should be removed from C++ standard or made sizeof 1, as in Android NDK. Let me explain…
Earlier quoted context omitted.
As long as you don’t care about errors, converting to UTF-8 is quite fast. Just use native calls: var decode = function (bytes) { return decodeURIComponent(escape(bytes)); } var encode = function (string) { return unescape(encodeURIComponent(string)); } (Definitely test it before wailing about benchmarks. My guess is that whatever else you’re doing is likely much slower.) If you do care about errors, or especially if…
I think you misunderstand -- I'm referring to the actual systems-level implementation of the browser engine itself. I'm not talking about the implementation of web apps. Consider a pattern like this: A page calls document.createElement(), adds a large text node (say, the collected works of Shakespeare in text form) to it, calls window.getComputedStyle() on that element, then throws the element away. This series of DO…
Earlier quoted context omitted.
Just use the term "string" to refer to utf-8, and the term "data in nonstandard encoding X" to refer to other encodings. In the article he puts in in terms of std::string, but more generally I think this is what he means.
You're confusing things. Strings cannot be utf-8 any more than you can be your signature. "strings" are abstract data structures. They are lists of characters. Not bytes, not integers, but characters. Often, we use the Unicode character set as the set of allowable characters. There are other character sets. Internally, strings often represent characters as integers. When using the Unicode character set, strings then…
Separate point: There is no such thing as an abstract string or integer in a computer, no matter what language you are using. Every string in a computer has an encoding - you have to store it as ones and zeros.
If we standardize on UTF-8 as an encoding, we just dont need to use the awkward phrase "UTF-8" in ordinary conversation.
Earlier quoted context omitted.
cat a b c There you go, a BOM in the middle of the file.
if the BOM appears in the middle of the file than it should be considered a zero width non breaking space (so basically ignored) so why is this an issue?