51–60 of 73 posts
Earlier quoted context omitted.
UTF-8 was plan9's invention, I don't think anybody saw it coming.
For more details: Rob Pike & Ken Thompson invented UTF-8 when prompted by X/Open (Not sure if X/Open and POSIX were already merged at that point) and they first implemented the proof of concept on plan9 ( https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt ).
I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps. It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They…
It's not as if they were the only ones. I had a lot of utf-related pain in the Linux terminal back in the day.
Vi and Emacs took a long time to handle variable length encodings well. My old Unix editor of choice, NEdit, never managed the switch at all.
I got a mail from my boss in utf-7 once. God knows how that happened.
I remember this hoopla. And windows wasn't the only victim of using "two bytes for a char" for unicode support. Java fell in this trap too. The team was split in deciding "whether to waste a single byte per character for string" -- that's in their words, and then decided to rather be ready for the future. That's in 1995. And then everyone later watched UTF-8 take over the world. The PHP team also wasted their version…
> Why didn't they go the UTF-8 round? The original Unicode marketing / position statement from 1988[1] may provide a clue: “In the Unicode system, a simple unambiguous fixed-length character encoding is integrated into a coherent overall architecture for text processing.” “Unicodes [sic] are the most straightforward multilingual generalization of ASCII codes: - Fixed length of character code (16 bits); [...]” “Are 16…
Earlier quoted context omitted.
What's depressing about the saga of utf8 adoption is that it's such an obvious solution. The ISO2022 standards used the principle since the 70s, and programmers of the era were widely accustomed to extending 8 bit instruction sets to multibyte instructions. It's one of those cases where you need to hit your head hard and repeatedly until you bite the bullet and accept that the cost of multibyte (for example, strlen()…
UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. This is why UTF8 wasn't universally hailed to begin with. Countries like Japan had already solved encoding differently while US was still using ASCII. (See JIS encoding). As time passed tho, the byte savings matter less. Also combined with transfer compression makes the encoding size differences n…
Earlier quoted context omitted.
UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. This is why UTF8 wasn't universally hailed to begin with. Countries like Japan had already solved encoding differently while US was still using ASCII. (See JIS encoding). As time passed tho, the byte savings matter less. Also combined with transfer compression makes the encoding size differences n…
> UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all…
I remember this hoopla. And windows wasn't the only victim of using "two bytes for a char" for unicode support. Java fell in this trap too. The team was split in deciding "whether to waste a single byte per character for string" -- that's in their words, and then decided to rather be ready for the future. That's in 1995. And then everyone later watched UTF-8 take over the world. The PHP team also wasted their version…
> Why didn't they go the UTF-8 round? The original Unicode marketing / position statement from 1988[1] may provide a clue: “In the Unicode system, a simple unambiguous fixed-length character encoding is integrated into a coherent overall architecture for text processing.” “Unicodes [sic] are the most straightforward multilingual generalization of ASCII codes: - Fixed length of character code (16 bits); [...]” “Are 16…
Earlier quoted context omitted.
> UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all…
Your argument uses a website whose code was written by English speakers. There would still be ASCII but verbose element names like "vector-page-toolbar-container" would definitely be shorter both in utf-8 and in utf-16 if they weren't written in English
Plus many identifiers come from libraries, and when creating their own identifiers many people use either full English or partial English no matter what language (it was a huge mistake to not use English for many identifiers in my first programming job, as you will invariably end up with a mishmash of two languages).
But it is easy enough to verify this with some actual websites: https://www.rakuten.co.jp is 330K in UTF-8 and 625K in UTF-16, https://ameblo.jp is 104K in UTF-8 and 187K in UTF-16, baidu.com is 360K in UTF-8 and 717K in UTF-16, sina.com.cn: 455K, 854K, daum.net: 666K, 1.2M.
And all of that is only the HTML document; if we'd add up the CSS – where there's almost no possibility to use non-ASCII outside of class and ID names – and JavaScript – where the filesize is usually dominated by React or jQuery or whatnot – things would skew even more in favour of UTF-8.
I'm sure there are examples where a page served over UTF-16 is smaller, such as pages with very little markup (like e.g. HN), but that is the common case, even for websites exclusively written and for users of CJK languages. Someone who does not speak a word of English will save many bytes of data every day with UTF-8. There's a reason all those websites are served over UTF-8 and not UTF-16.
But for the sake of the argument, let's replace all class="...", id="..", and data-event-name=".." with strings of the same length consisting of "回". That grows the filesize from 118K to 151K, and ... it's still larger in UTF-16 with 207K. We could start replacing more stuff and eventually UTF-16 may win, maybe. But you have to use a lot of CJK. Let's use a random excerpt:
UTF-8 91×1 + 60×3 = 271 bytes
UTF-16 91×2 + 60×2 = 302 bytes
And to repeat, there are certainly cases where UTF-16 is smaller. Markdown documents and other plain text files is an obvious one, but HTML is rarely one of them.
But imagine actually checking things before making a claim...
Earlier quoted context omitted.
UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. This is why UTF8 wasn't universally hailed to begin with. Countries like Japan had already solved encoding differently while US was still using ASCII. (See JIS encoding). As time passed tho, the byte savings matter less. Also combined with transfer compression makes the encoding size differences n…
> UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all…
Earlier quoted context omitted.
> Why didn't they go the UTF-8 round? The original Unicode marketing / position statement from 1988[1] may provide a clue: “In the Unicode system, a simple unambiguous fixed-length character encoding is integrated into a coherent overall architecture for text processing.” “Unicodes [sic] are the most straightforward multilingual generalization of ASCII codes: - Fixed length of character code (16 bits); [...]” “Are 16…
Just wanted to say thank you for linking these documents! I find the history of character encoding design so interesting, especially from primary sources such as these. :)
[1] https://apps.dtic.mil/sti/citations/ADA221614 (scanned PDF) or http://archive.adaic.com/pol-hist/history/9x-history/reports... (PostScript) or http://archive.adaic.com/pol-hist/history/9x-history/reports... (ASCII)
Earlier quoted context omitted.
> UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all…
Is there any particular reason to think HTML would be high on the list of things they thought about when they decided this? It's not like the world wide web had taken over everything yet…
For example plain text emails still have considerable ASCII data in the headers, no matter which language you use to send them, and depending on the length of your email UTF-8 may be smaller than UTF-16. I just checked a very simple email in my inbox, and it has 165 lines of headers for 21 lines of actual email body text (plain email, no MIME). Even after removing the more modern headers (DKIM, X-Spam-, etc.) we're still left with 35 lines, or 1,886 bytes. The actual email text is 765 bytes, in basic English. If the email text was in CJK it would still be slightly over 1K smaller* in UTF-8 (4,181 bytes vs. 5,302).
Longer emails would be smaller in UTF-16. I don't know what the average works out to, but ~21 lines seems about average for an email, give or take.