Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

Post reply on HN
Print view View original

11–20 of 159 posts

Re: Unicode Is Awesome

#11

Post by flohofwoe »

IMHO the article should mention that UTF-16 was (more or less) a hack to fix Windows and some other systems which didn't see the light and use UTF-8 from the start. UTF-16 has all the disadvantes of UTF-8 (variable length) and UTF-32 (endianess), but none of the advantages (encoding as endian-agnostic, 7-bit ASCII compatible byte stream like UTF-8, or a fixed-width encoding like UTF-32). UTF-16 should really be considered a hack to talk to (mainly) Windows APIs.

Also, obligatory link to: https://utf8everywhere.org/

Re: Unicode Is Awesome

#12

Post by zzo38computer »

euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

I think Unicode is terrible. Remove everything. Use ASCII and other character sets.

Unicode is OK for searching for data using many different languages (if you omit much of the junk such as emoji and compatibility characters), although might not be best with that too.

You can't effectively use one character set well for everything; different applications have different requirements. Unicode is equally bad for everything, rather than e.g. ASCII which is good for some stuff and not usable for some stuff, and other character set which is a similar thing. Many things you just can't do accurately with Unicode.

Re: Unicode Is Awesome

#14

Post by maxerickson »

euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

25 or 50 examples of inconsistencies would help support your tone.

Re: Unicode Is Awesome

#15

Post by RcouF1uZ4gsC »

> Unicode is simply a 16-bit code - Some people are under the misconception that Unicode is simply a 16-bit code where each character takes 16 bits and therefore there are 65,536 possible characters. This is not, actually, correct. It is the single most common myth about Unicode, so if you thought that, don't feel bad.

Verity Stob has a great column

https://www.theregister.co.uk/2013/10/04/verity_stob_unicode...

where she says that it is wrong to call this a myth, since that was how it was originally designed. It is better characterized as being obsolete, rather than a myth.

Re: Unicode Is Awesome

#16

Post by dfox »

flohofwoe wrote: post #11

IMHO the article should mention that UTF-16 was (more or less) a hack to fix Windows and some other systems which didn't see the light and use UTF-8 from the start. UTF-16 has all the disadvantes of UTF-8 (variable length) and UTF-32 (endianess), but none of the advantages (encoding as endian-agnostic, 7-bit ASCII compatible byte stream like UTF-8, or a fixed-width encoding like UTF-32). UTF-16 should really be consi…

The main point that should be emphasised is that any encoding with fixed size unicode codepoints is mostly unnecessary as you mostly don’t care about the codepoints but about how the resulting glyphs or even glyph runs look like.

My experience is that if you want to implement efficient unicode-aware text editor then the right datastructure is list of lines and you have to simply forget about gap buffers, ropes and what not (unless you really care about 32k+ lines/paragraphs, which is when rope-style representation starts to make sense as long as the breaks match unicode semantics)

Re: Unicode Is Awesome

#17

Post by wvenable »

flohofwoe wrote: post #11

IMHO the article should mention that UTF-16 was (more or less) a hack to fix Windows and some other systems which didn't see the light and use UTF-8 from the start. UTF-16 has all the disadvantes of UTF-8 (variable length) and UTF-32 (endianess), but none of the advantages (encoding as endian-agnostic, 7-bit ASCII compatible byte stream like UTF-8, or a fixed-width encoding like UTF-32). UTF-16 should really be consi…

Windows and many other operating systems and languages (Java) got on board with Unicode back when the character set would fit in 16bits. The character set originally used was UCS-2 (not UTF-16). UTF-16 came next to extend the Unicode character set beyond 65536 code points.

UTF-8 wasn't even invented until well after all these operating systems and languages deployed Unicode.

They didn't see the light of day to use UTF-8 because they didn't have a time machine to make that possible.

Re: Unicode Is Awesome

#18

Post by flohofwoe »

zzo38computer wrote: post #12
euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

I think Unicode is terrible. Remove everything. Use ASCII and other character sets. Unicode is OK for searching for data using many different languages (if you omit much of the junk such as emoji and compatibility characters), although might not be best with that too. You can't effectively use one character set well for everything; different applications have different requirements. Unicode is equally bad for everyth…

Well if you write an application for a 'non-technical' international audience, you'll have to support international text output. And representing text as one of the universal Unicode encodings is still much better than the codepage mess and region-specific multi-byte encodings like Shift-JIS we had before.

UTF-8 is usually the best choice both for simple tools and 'user-facing applications' since it is backward-compatible with 7-bit ASCII (e.g. usually you don't need to change a thing in your code, at least if you just pass strings around).

If you encounter a byte in an UTF-8 encoded string which has the topmost bit cleared, it's an ASCII character and definitely not part of a multi-byte sequence. If the topmost bit is set, the byte is part of a multi-byte-sequence, and such sequences must remain intact.

Re: Unicode Is Awesome

#19

Post by WorldMaker »

akdor1154 wrote: post #13

Unicode is great, emoji are a (technically impressive) monstrosity.

Emoji are an almost critical Unicode democratizing need. They aren't doing anything that other languages encoded with Unicode don't already do (and haven't already done since the beginning of Unicode). This article itself points out several key existing relatives, such as how Arabic, one of the most common and important written languages in the world, or the very important CJK family of written languages, used ZWJ and ZWNJ well before Emoji made it "cool" to other parts of the world, most especially the English-writing contingent that has long thought of Unicode as simply "ASCII plus a bunch of other stuff I might never use". Suddenly a lot of English documents have embedded emoji that deeply matters to the writers, and there are fewer excuses to treat Unicode as "ASCII+" and more cases where doing so is not only wrong (broken surrogate pairs, incorrect codepoint analysis for ZWJ, etc), but very visibly wrong in a way that users care and complain about it.

Re: Unicode Is Awesome

#20

Post by mr_toad »

jcranmer wrote: post #10
euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accuratel…

Most resumes I’ve seen don’t even bother with the accents.

Most are written in Word on Windows, and I’d guess that most people don’t even know how to access the accented characters.

Post reply on HN