Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

Post reply on HN
Print view View original

91–100 of 159 posts

Re: Unicode Is Awesome

#91

Post by msla »

zzo38computer wrote: post #31
flohofwoe wrote: post #18

Earlier quoted context omitted.

Well if you write an application for a 'non-technical' international audience, you'll have to support international text output. And representing text as one of the universal Unicode encodings is still much better than the codepage mess and region-specific multi-byte encodings like Shift-JIS we had before. UTF-8 is usually the best choice both for simple tools and 'user-facing applications' since it is backward-compa…

UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text). But I was not talking about the encoding; I was talking about the Unicode character set.

> UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text).

This is utterly incoherent.

Re: Unicode Is Awesome

#92

Post by lifthrasiir »

cryptonector wrote: post #83
jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

If we could go back in time to Unicode's beginning and start over but with all that we know today... Unicode would still look a lot like what it looks like today, except that: - UTF-8 would have been specified first - we'd not have had UCS-2, nor UTF-16 - we'd have more than 21 bits of codespace - CJK unification would not have been attempted - we might or might not have pre-composed codepoints[0] - a few character-s…

> - we'd have more than 21 bits of codespace

Just in case, there is an extension to Unicode [1] if we would ever be out of code points.

[1] http://ucsx.org/

Re: Unicode Is Awesome

#93

Post by cryptonector »

lifthrasiir wrote: post #92
cryptonector wrote: post #83

Earlier quoted context omitted.

If we could go back in time to Unicode's beginning and start over but with all that we know today... Unicode would still look a lot like what it looks like today, except that: - UTF-8 would have been specified first - we'd not have had UCS-2, nor UTF-16 - we'd have more than 21 bits of codespace - CJK unification would not have been attempted - we might or might not have pre-composed codepoints[0] - a few character-s…

> - we'd have more than 21 bits of codespace Just in case, there is an extension to Unicode [1] if we would ever be out of code points. [1] http://ucsx.org/

The answer is to deprecate and obsolete UTF-16, then remove the artificial 21 bit limit on UTF-8. Or remove the artificial 21 bit limit on UTF-8 and let UTF-16 just die. Or... see below.

There's an essential, very necessary ingredient: a sense of urgency at MSFT and various JavaScript implementors (and others) to ditch UTF-16.

Yet legacy is forever, so I wouldn't expect UTF-16 to die.

Most likely, when we run out of codespace we'll see UC assign a "UTF-16 SUX, JUST UPGRADE TO UTF-8 ALREADY" codepoint that newer codepoints can be mapped to when converting to UTF-16, then lift the 21 bit limit on UTF-8.

Re: Unicode Is Awesome

#94

Post by lifthrasiir »

Years ago I've posted support material [1] for Hangul filler mentioned in the article, reproduced below:

---

U+3164 HANGUL FILLER is one of the stupidest choices made by character sets. Hangul is noted for its algorithmic construction and Hangul charsets should ideally be following that. Unfortunately, the predominant method for multibyte encoding was ISO 2022 and EUC and both required a rather small repetoire of 94 × 94 = 8,836 characters [0] which are much less than required 19 × 21 × 28 = 11,172 modern syllables.

The initial KS X 1001 charset, therefore, only contained 2,350 frequent syllables (plus 4,888 Chinese characters with some duplicates, themselves becoming another Unicode headache). Notwithstanding the fact that remaining syllables are not supported, this resulted in a significant complexity burden for every Hangul-supporting software and there were confusion and contention between KS X 1001 and less interoperable "compositional" (johab) encodings before Unicode. The standardization committee has later acknowledged the charset's shortcoming, but only by adding four-letter (thus eight-byte) ad-hoc combinations for all remaining syllables! The Hangul filler is a designator for such combinations, e.g. `ㄱㅏ` denotes `가` and `ㅂㅞㄺ` denotes `뷁` (not in KS X 1001 per se).

Hangul filler was too late in the scene that it had virtually no support from software industry. Sadly, the filler was there and Unicode had to accept it; technically it can be used to designate a letter (even though Unicode does not support the combinations) so the filler itself should be considered as a letter as well. What, the, hell.

[0] It is technically possible to use 94 × 94 × 94 = 830,584 characters with three-byte encoding, but as far as I know there is no known example of such charset designed (thus no real support too).

---

I should also mention that early Mozilla (and thus Firefox) had once supported ad-hoc combinations for KS X 1001, got interoperability problems and dropped the support later. Nowadays we treat KS X 1001 as an alias of Windows code page 949 for the sake of compatibility [2].

[1] https://github.com/Wisdom/Awesome-Unicode/issues/4

[2] https://encoding.spec.whatwg.org/#index-euc-kr

Re: Unicode Is Awesome

#95

Post by lifthrasiir »

cryptonector wrote: post #93
lifthrasiir wrote: post #92

Earlier quoted context omitted.

> - we'd have more than 21 bits of codespace Just in case, there is an extension to Unicode [1] if we would ever be out of code points. [1] http://ucsx.org/

The answer is to deprecate and obsolete UTF-16, then remove the artificial 21 bit limit on UTF-8. Or remove the artificial 21 bit limit on UTF-8 and let UTF-16 just die. Or... see below. There's an essential, very necessary ingredient: a sense of urgency at MSFT and various JavaScript implementors (and others) to ditch UTF-16. Yet legacy is forever, so I wouldn't expect UTF-16 to die. Most likely, when we run out of…

The proposed UCS-G-8 encoding [1] does exactly that. And in case UTF-16 never dies, the website also proposes extensions to UTF-16 (and UTF-32) as well.

[1] http://ucsx.org/g8

Re: Unicode Is Awesome

#96

Post by lifthrasiir »

happytoexplain wrote: post #46
paulddraper wrote: post #43

Earlier quoted context omitted.

Also a law that the author is thinking of one and only one programming language. > String length is typically determined by counting codepoints. That depends entirely on what "strings" you are talking about. In C/Go/Rust/Ruby, char*/string/std::string::String/String is bytes. In Java/JavaScript, java.lang.String/String is UTF-16 code units. In Python 3, str is code points. In Swift, String is extended grapheme cluste…

I'm in love with Swift's approach, where the default representation is a well defined thing that both users and developers think of as "characters", but all the other representations are trivially accessible.

I disagree. Grapheme clusters are locale-dependent, much like string collation is locale-dependent. What Unicode gives you by default, the (extended) grapheme cluster, is as useful as the DUCET (Default Unicode Collation Element Table); while you can live with them, you would be unsatisfied. In fact there are tons of Unicode bugs that can't be corrected due to the compatibility reason, and can only be fixed via tailored locale-dependent schemes.

I would like to avoid locales in the language core. It would be great to have locale stuffs in the standard library, but without locale information you can't treat strings as (human) texts.

Re: Unicode Is Awesome

#97

Post by ken »

jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

The Consortium chose the domain of the problem space, though. "Text" could be either much simpler or much more complex than Unicode chose to model it. They picked what features they wanted to support, and now we all have to live with that.

In my lifetime, I've seen text systems that choose to include style (bold, italic), color (fore and back), or size with each character. Unicode did not (generally) choose to include those, even though they're arguably part of "global text". Ligatures, too, are generally considered the domain of font rendering, not text storage. Vertical text was, too, until a few months ago. "Historical" writing directions are apparently still considered out of scope for Unicode. Linear A is in scope, though, even though nobody is sure what the characters mean.

Unicode did choose to be backwards compatible with ASCII, and include most of the crazy PETSCII characters (which were pretty popular for just a couple years but not really "global text"), and some mathematical symbols that no mathematician had a use for.

They chose to include both a nice component-combining system and also pre-combined glyphs where legacy codepages had used them. They chose to implement Han unification, but not analogous unifications across other scripts which have even more similar glyphs.

I've dug into the details of Unicode since 3.0 (20 years ago!), and found it's full of arbitrary decisions and legacy workarounds. The contributors are smart but the result looks the same as any committee full of people with conflicting goals.

Legacy support is why it's so well-adapted, and I've never seen a system where piling on legacy support made it "well-designed".

Suppose you wanted to make a system for "universal computation". The analogous method would have been to take Win16, Win32, Mac OS 9, Mac OS X, Linux, and Solaris, define the superset of all of their features, and invent a binary format which supported all of it natively. Legacy support might help get it adopted faster but nobody would call it well-designed. 20 years later, it'd clearly be simultaneously too weak and too powerful for all types of computation we want to do.

Unicode is an amazing political accomplishment. Technically, it seems rather mediocre. Nobody would ever design a text system like this unless held back by mountains of legacy 1970's/1980's systems.

Re: Unicode Is Awesome

#98

Post by kazinator »

cryptonector wrote: post #85
kazinator wrote: post #9

Earlier quoted context omitted.

No, where Unicode is complicated is where the Unicode people decided to make it complicated to bolster their egos, to the detriment of everyone downstream of them. Like with most standardization, the people at the helm are the wrong people with the wrong motivations.

You're demonstrably wrong. Most complexity in Unicode derives from: - real complexity in human scripts - politics neither of which is something that Unicode could have avoided. Complexity in human scripts necessarily leads to complexity in Unicode. Not having Unicode at all would be much worse than Unicode could possibly seem to you -- you'd have to know the codeset/encoding of every string/file/whatever, and never l…

I didn't explain myself earlier because I knew that apologies of this sort would be coming, which would make it unnecessary.

Unicode is rot, and you seem to be well-versed in the reasons why.

You're not convincing because you're not convinced.

Re: Unicode Is Awesome

#99

Post by rimliu »

zzo38computer wrote: post #12
euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

I think Unicode is terrible. Remove everything. Use ASCII and other character sets. Unicode is OK for searching for data using many different languages (if you omit much of the junk such as emoji and compatibility characters), although might not be best with that too. You can't effectively use one character set well for everything; different applications have different requirements. Unicode is equally bad for everyth…

    > Use ASCII and other character sets.
We have tried that before. It did not work, and it was not pretty. You may not know, but there is a huge demand to be able to use characters from different sets in the same document. How do you do Wikipedia without Unicode? (E.g. this: Alexander Sergeyevich Pushkin (English: /ˈpʊʃkɪn/;[1] Russian: Александр Сергеевич Пушкин[note 1]).

How would you implement any chat/messaging app for the international audience? Like in my current company, I am sure at least five languages, each with its own alphabet, are used to communicate in Slack.

For me, app not supporting Unicode is broken.

Re: Unicode Is Awesome

#100

Post by deathanatos »

notatoad wrote: post #86
deathanatos wrote: post #22

I swear there should be some rule or law about how Unicode articles will inevitably muddle code units/points / grapheme clusters / bytes together. > String length is typically determined by counting codepoints. > This means that surrogate pairs would count as two characters. If you were counting code points, a surrogate pair would be 1. If it's two, you're counting code units. > Combining multiple diacritics may be s…

>I swear there should be some rule or law... Now is your chance! Distill this comment down into something pithy and deathanatos' law could be a thing.

Naming things is the hardest problem ;)
Post reply on HN