Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

151–160 of 289 posts

Re: UTF-8 Everywhere

#151

Post by ftvy »

pjscott wrote: post #101
anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

Unicode is complicated because the languages it needs to handle are, alas, complicated. UTF-8 is super simple. It's a variable-length encoding for 21-bit unsigned integers. Wikipedia gives a handy table showing how it works: https://en.wikipedia.org/wiki/UTF-8#Description

When I wrote a very primitive UTF-8 library, I really began to appreciate UTF-8's design. For example; the first byte says how many bytes the character requires. At first it was daunting, but when I put 2 and 2 together, it really opened up.

I am sure there are many aspects I am missing about UTF-8, but it is all reasonable in its design and implementation.

For reference, I was converting between code points and actual bytes, and also implemented strlen and strcmp (which for the latter the standard library apparently handles fine).

Re: UTF-8 Everywhere

#152

Post by cygx »

klodolph wrote: post #111
shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”…

Han unification was pushed through by western interests, by my understanding.

Note that as far as I'm aware, the interest in question was the initial 16-bit limit of the character set and later on the non-proliferation of competing standards.

Also note that while Han unification is the most prominent example, there are technically similar cases, which just aren't as charged culturally. For one, Unicode doesn't encode German Fraktur: While some characters are available due to their use in mathematics, it's lacking the corresponding variants of ä, ö, ü, ß, ſ as well as specific ligatures. So if you want to intermix modern with old German writing, you'll also have to go out-of-band.

Re: UTF-8 Everywhere

#153

Post by GnarfGnarf »

I came to the same conclusion years ago. My app is Win32, but I never defined UNICODE or used the TCHAR abomination. All strings are stored as UTF8 until they are passed to Win32 APIs, whereupon they are converted to UCS-2. I explicitly call the wchar version of functions (ex: TextOutW). This strategy enabled me to transition easily and safely from single-byte ASCII (Windows 3.1) to Unicode.

The database is also UTF8.

Re: UTF-8 Everywhere

#154

Post by crazygringo »

droopyEyelids wrote: post #143
cryptonector wrote: post #123

Earlier quoted context omitted.

There are tens of thousands of characters in all the human scripts. If you're a librarian, scholar, researcher -- why would you not want to be able to use them seamlessly??

If there was a complicated tool that claimed it could do the job of every tool in history, or a simple tool that was focused to cover 99% of the work you do-- and we lived on planet earth-- which would you choose?

Umm... but ASCII doesn't work for 99% of people's work.

A majority of the world's population have writing systems that ASCII doesn't encode.

So not really sure what you're suggesting here.

Re: UTF-8 Everywhere

#155

Post by jeltz »

camgunz wrote: post #116
jeltz wrote: post #86

Earlier quoted context omitted.

But is it really a plurality? Portuguese, English, Spanish, Turkish, Vietnamese, French, Indonesian and German are stored more efficiently in UTF-8 while Chinese, Korean and Japanese are stored less effeciently. My gut feel is that more people use the Latin script than people using CJK scripts. Indic scripts, Thai, Cyrillic, etc are stored using two bytes in both UTF-8 AND UTF-16. And thus ignores markup which is in…

Looking at the basic multilingual plane [1], UTF-8 will use > 2 bytes to encode essentially anything that isn't: * ASCII/Latin * Cyrillic * Greek * Most of Arabic That leaves out: * China * India * Japan * Korea * All of Southeast Asia Re: markup, think about any text that's in a database, stored in RAM, or stored on a disk--relatively little of it will be in noisy ASCII markup formats like HTML or XML. [1]: https://…

> All of Southeast Asia

Did you forget Indonesia, Vietnam, Malaysia, Brunei and the Philippines?

Re: UTF-8 Everywhere

#156

Post by crazygringo »

droopyEyelids wrote: post #141
jaseemabid wrote: post #138

Earlier quoted context omitted.

ASCII is English and limiting access to knowledge for the rest of humanity for a simpler encoding is just not an acceptable option. Someone needs to interpret those 7k words and write a (complicated?) program once so that billions can read in their own language? Sounds like an easy win to me.

counterpoint: A complicated program is never an easy win, and English is already spoken in every country in the world.

So your argument is... it's easier to teach billions of people fluent English... than for software to support UTF-8?

You are aware that a majority of the world's population speaks no English whatsoever?

Re: UTF-8 Everywhere

#157

Post by kllrnohj »

heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

> It is a pain in the ass to have a variable number of bytes per char.

This is from API & language mistakes more than an issue with UTF-8 itself.

If you actually design your API & system around being UTF-8, like Rust did, then there's really no issue for the programmer. The API enforces the rules, and still gives you things like a simple character iterator (with characters being 32-bit, so that it actually fits: https://doc.rust-lang.org/std/char/index.html). The String class handles all the multi-byte stuff for you, you never "see" it: https://doc.rust-lang.org/std/string/struct.String.html

Retrofitting this into existing languages isn't going to be easy, but that's not an excuse to not do it at all, either.

Re: UTF-8 Everywhere

#158

Post by DagAgren »

kazinator wrote: post #96
DagAgren wrote: post #54

Earlier quoted context omitted.

You can't even write proper English in ASCII. ASCII is an absolute dead end. It's history. Actually representing human language is HARD. It is also absolutely necessary. Whatever solution you choose is going to be complicated, because it is solving a very complicated problem. Throwing your hands up and going "oh this is too hard, I don't like it" will get you nowhere.

You can't write proper snooty English in ASCII, with diaereses and whatnot.

ASCII doesn't have have all the punctuation regularly used in English.

Re: UTF-8 Everywhere

#159

Post by ignoramous »

TheCoelacanth wrote: post #120
anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

You only need one sentence to explain why ASCII isn't sufficient: There are languages other than English.

> You only need one sentence to explain why ASCII isn't sufficient

Nitpick: ASCII is sufficient when you consider that Base64, despite its 33% overhead from representing 6 bits with 8 bits, makes life easier for certain classes of software.

Re: UTF-8 Everywhere

#160

Post by camgunz »

jeltz wrote: post #155
camgunz wrote: post #116

Earlier quoted context omitted.

Looking at the basic multilingual plane [1], UTF-8 will use > 2 bytes to encode essentially anything that isn't: * ASCII/Latin * Cyrillic * Greek * Most of Arabic That leaves out: * China * India * Japan * Korea * All of Southeast Asia Re: markup, think about any text that's in a database, stored in RAM, or stored on a disk--relatively little of it will be in noisy ASCII markup formats like HTML or XML. [1]: https://…

> All of Southeast Asia Did you forget Indonesia, Vietnam, Malaysia, Brunei and the Philippines?

Again, here's what UTF-8 will use Basic Latin (Lower half of ISO/IEC 8859-1: ISO/IEC 646:1991-IRV aka ASCII) (0000–007F)

Latin-1 Supplement (Upper half of ISO/IEC 8859-1) (0080–00FF)

Latin Extended-A (0100–017F)

Latin Extended-B (0180–024F)

IPA Extensions (0250–02AF)

Spacing Modifier Letters (02B0–02FF)

Combining Diacritical Marks (0300–036F)

Greek and Coptic (0370–03FF)

Cyrillic (0400–04FF)

Cyrillic Supplement (0500–052F)

Armenian (0530–058F)

Aramaic Scripts:

    Hebrew (0590–05FF)

    Arabic (0600–06FF)

    Syriac (0700–074F)

    Arabic Supplement (0750–077F)

    Thaana (0780–07BF)

    N'Ko (07C0–07FF)
In UTF-8, everything over U+0800 requires > 2 bytes. Am I misunderstanding something? It's possible.
Post reply on HN