Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

Post reply on HN
Print view View original

111–120 of 159 posts

Re: Unicode Is Awesome

#111

Post by tialaramex »

kps wrote: post #81
jakeogh wrote: post #75

What's the code point for uppercase superscript Z?

There isn't one. Unicode considers superscripting a matter of presentation, which Unicode doesn't cover, except when it does.

More particularly: Presentation variant is not a justification for inclusion in Unicode BUT prior encoding in another character set is.

Unicode sets a high priority on roundtripping. The idea is that if you take some data in any one character set X and convert it to Unicode, you should preserve all the meaning by doing this, such that you could losslessly convert it back to encoding X.

It's like the wordprocessor problem where users say they only want 10% of the features of a popular wordprocessor but it turns out each user wants a different 10% and so the only way to deliver what they all want is to deliver 100% of the features. Likewise, Unicode has all the weird features of every legacy character set which was embraced BUT it doesn't arbitrarily add new weird features, although you could argue that some of the work done for Unicode has that effect e.g. the way flags work or the Fitzpatrick modifiers.

If Unicode had insisted upon never encoding anything that might be a presentation feature, it'd be a long forgotten academic project that never went anywhere and we'd all be using some (probably Microsoft designed) 16-bit ASCII superset today.

Re: Unicode Is Awesome

#112

Post by v8engine »

lifthrasiir wrote: post #95
cryptonector wrote: post #93

Earlier quoted context omitted.

The answer is to deprecate and obsolete UTF-16, then remove the artificial 21 bit limit on UTF-8. Or remove the artificial 21 bit limit on UTF-8 and let UTF-16 just die. Or... see below. There's an essential, very necessary ingredient: a sense of urgency at MSFT and various JavaScript implementors (and others) to ditch UTF-16. Yet legacy is forever, so I wouldn't expect UTF-16 to die. Most likely, when we run out of…

The proposed UCS-G-8 encoding [1] does exactly that. And in case UTF-16 never dies, the website also proposes extensions to UTF-16 (and UTF-32) as well. [1] http://ucsx.org/g8

As a lay-developer, I know unicode is what you need for international character support. Oh, so what are the options: utf-8, utf-16, utf-32. I will choose utf-32 just because 32 > 16 or 8.

Re: Unicode Is Awesome

#113

Post by tasogare »

jcranmer wrote: post #72
sansnomme wrote: post #63

Earlier quoted context omitted.

Yeah, that is quite inconsistent. Kanji literally means "Chinese Character" so it should be the same for the letter A. Unless a French A isn't equivalent to an English A.

Arabic numerals (0123456789) are not to be confused with the Arabic numerals (٠١٢٣٤٥٦٧٨٩). So the fact that kanji literally means "Chinese character" doesn't mean that kanji and hanzi should be considered the same script. The Latin script (that which I write right now) and the Cyrillic script both derived heavily from the Greek script, especially the capital letters--fully 60% of them are identical in Latin and Greek…

> So the fact that kanji literally means "Chinese character" doesn't mean that kanji and hanzi should be considered the same script.

The Japanese writing system differs from the Chinese one by having its own distincts scripts (hiragana, katakana), but most of its subset made of Chinese characters (kanji) is the same than the Chinese script. The most comprehensive Chinese character dictionary is a Japanese one (Daikanwa jiten), which give definition and Japanese readings and this is possible precisely because the script is the same.

The only differences are characters created for use in Japan (kokuji) which can be treated as an extension like the Vietnamese Nôm, characters simplified by the Japanese government (some jôyô kanji) and variation in some glyph's shape (黃/黄). So, treating the full inventory of these languages as different scripts wouldn't make more sense than encoding the English, French and Czech alphabets separately because few characters differ.

My opinion is that Han unification makes sense, but the mistake made was to encode the variant interpretation at application level (e.g. html lang tags), which is not portable. I don't know how Unicode variant form works in details (putting a trailing code to a character to indicate precisely which variant is meant) but something like that at text encoding level could ease a lot of pain.

Re: Unicode Is Awesome

#114

Post by bonoboTP »

GrantSolar wrote: post #108
bonoboTP wrote: post #89

Earlier quoted context omitted.

I don't see anything special in the linked image. The As look as they would in Latin script. (probably not what you meant, but just in case: the fourth letter is not a Cyrillic A but a D.)

I think the image is not meant to show the problem but show a case where if the Cyrillic A had been stylised the same way that the English A is in the English version, the two distinct letters would become indistinguishable such that the Cyrillic title would effectively read "The Mlndlloriln"

I see, for those out of the loop, the English title screen does not have the horizontal bar in the As.

Re: Unicode Is Awesome

#115

Post by bawolff »

rimliu wrote: post #99
zzo38computer wrote: post #12

Earlier quoted context omitted.

I think Unicode is terrible. Remove everything. Use ASCII and other character sets. Unicode is OK for searching for data using many different languages (if you omit much of the junk such as emoji and compatibility characters), although might not be best with that too. You can't effectively use one character set well for everything; different applications have different requirements. Unicode is equally bad for everyth…

> Use ASCII and other character sets. We have tried that before. It did not work, and it was not pretty. You may not know, but there is a huge demand to be able to use characters from different sets in the same document. How do you do Wikipedia without Unicode? (E.g. this: Alexander Sergeyevich Pushkin (English: /ˈpʊʃkɪn/;[1] Russian: Александр Сергеевич Пушкин[note 1]). How would you implement any chat/messaging app…

Wikipedia didnt use unicode originally, en, da, sv, nl language wikipedia all used windows-1252. This all changed somewhere around 2004 i think, but there is still legacy code to deal with edits from before the switchover point.

I imagine the answer is, it kind of sucked but people made due the best they could with the limited allowed characters. Its not like IPA notation is a critical feature

Re: Unicode Is Awesome

#116

Post by nabla9 »

Unicode has two really great features.

* It names and defines things and sets standard. This seems trivial but is incredibly useful.

* Unicode encodings, mainly UTF-8 are good storage format for text (as a data structure for editing text, not so much if you want to be universal).

Unicode has one really horrible failing.

The 'user-perceived character' (Unicode terminology) is arguably the most important unit in text. Unicode approximates user-perceived characters using set of general rules to define grapheme clusters. A Grapheme cluster is a sequence of adjacent code points that should be treated as a unit by applications. Unfortunately the ruleset and definition is inadequate. Sometimes you need two grapheme clusters to define one unit.

If you get UTF-8 encoded and normalized string from somewhere from some unspecified time and era, don't know what application wrote it, using what version of UNICODE standard and what was the locale, you may lose some information.

Unicode should have added explicit encoding for user-perceived character boundaries (either fixed grapheme cluster eoncoding or completely different encoding). Let the writing software define it explicitly. It would have been future-proof (new software in the future can understand old strings) and past-proof (ancient software can understand and edit strings written in the future).

Re: Unicode Is Awesome

#117

Post by rswail »

kazinator wrote: post #98
cryptonector wrote: post #85

Earlier quoted context omitted.

You're demonstrably wrong. Most complexity in Unicode derives from: - real complexity in human scripts - politics neither of which is something that Unicode could have avoided. Complexity in human scripts necessarily leads to complexity in Unicode. Not having Unicode at all would be much worse than Unicode could possibly seem to you -- you'd have to know the codeset/encoding of every string/file/whatever, and never l…

I didn't explain myself earlier because I knew that apologies of this sort would be coming, which would make it unnecessary. Unicode is rot, and you seem to be well-versed in the reasons why. You're not convincing because you're not convinced.

And the alternative would be what exactly?

Re: Unicode Is Awesome

#118

Post by arrrg »

ken wrote: post #97
jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

The Consortium chose the domain of the problem space, though. "Text" could be either much simpler or much more complex than Unicode chose to model it. They picked what features they wanted to support, and now we all have to live with that. In my lifetime, I've seen text systems that choose to include style (bold, italic), color (fore and back), or size with each character. Unicode did not (generally) choose to includ…

I think you fundamental mistake is thinking that you can separate technical and political concerns or that those two things really are always distinct and cleanly separable.

To me having that kind of discussion really doesn’t make a lot of sense. You know, since “we live in a society” (at the risk of quoting a clearly thought-terminating cliche).

Unicode contains within itself thousands of design-decisions, many of them trade-offs. After the fact it’s always extremely easy to swoop in and nitpick those trade-offs. No possible world exists where all those trade-offs are made correctly and what’s more defining what a “correct” trade-off even is is frequently simply impossible to know.

(Just one example to illustrate the scope of this problem: A certain trade-off might be worth making in one direction for use case A and in another direction for use case B, however it’s not really easy to find out whether use case A or use case B are more frequent in the wild. What if both use cases are about equally as important? Now imagine the trade-off space not being a binary space but multi-dimensional. Now imagine not just two but several use cases. Now imagine use case usage changing over time and new use cases emerging in the future.)

Re: Unicode Is Awesome

#119

Post by paulddraper »

cryptonector wrote: post #83
jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

If we could go back in time to Unicode's beginning and start over but with all that we know today... Unicode would still look a lot like what it looks like today, except that: - UTF-8 would have been specified first - we'd not have had UCS-2, nor UTF-16 - we'd have more than 21 bits of codespace - CJK unification would not have been attempted - we might or might not have pre-composed codepoints[0] - a few character-s…

    - Old grapheme clusters and new extended grapheme clusters would be the same.

Re: Unicode Is Awesome

#120

Post by jakeogh »

tialaramex wrote: post #111
kps wrote: post #81

Earlier quoted context omitted.

There isn't one. Unicode considers superscripting a matter of presentation, which Unicode doesn't cover, except when it does.

More particularly: Presentation variant is not a justification for inclusion in Unicode BUT prior encoding in another character set is. Unicode sets a high priority on roundtripping. The idea is that if you take some data in any one character set X and convert it to Unicode, you should preserve all the meaning by doing this, such that you could losslessly convert it back to encoding X. It's like the wordprocessor pro…

Is there a realitvely easy way to find the character set that was included for uppercase superscript W? (ᵂ)
Post reply on HN