Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

Post reply on HN
Print view View original

71–80 of 159 posts

Re: Unicode Is Awesome

#71

Post by TheDong »

jcranmer wrote: post #59
TheDong wrote: post #33

Earlier quoted context omitted.

There's a not-insignificant number of Japanese websites that can only correctly display using EUC-JP or ShiftJIS. It seems very Latin/ASCII centric to push for disabling non-UTF-8 encodings, especially since the only reason UTF-8 works so well on ASCII websites is due to its backwards compatibility. If it were the reverse, and UTF-8 were backwards compatible with EUC-JP/CJK, but not ASCII, I doubt you'd be pushing fo…

There is no character in EUC-JP or Shift-JIS that is not in Unicode--the explicit goal of Unicode in its original formulation was to be able to losslessly round-trip any other charset through Unicode, and the initial version of Unicode incorporated the source kanji lists for the EUC-JP/Shift-JIS charsets in their entirety.

That's true, but you misunderstood what I meant.

The parent comment seemed to be implying that we should drop support for non-utf8 charsets.

To me, that rings like saying a website with 'charset=EUC-JP' (such as http://www.os2.jp/) should be broken, as in browsers should error out or display a large quantity of black boxes due to it using a non-utf-8 encoding.

I'm claiming the only reason the author thinks that's really viable is because in our western-centric world, we see mostly ascii and utf8. Things that, if you flip to only utf-8, both still look fine.

CJK websites, on the other hand, that are using the equivalent of ASCII will have to be manually upgraded to display correctly if browsers drop their support.

Sure, all their characters can be represented in utf-8, but there's large swathes of websites that will never be updated to a new charset, and it's only a western-centric view that can so blithely suggest breaking them all.

Re: Unicode Is Awesome

#72

Post by jcranmer »

sansnomme wrote: post #63
ncmncm wrote: post #48

Earlier quoted context omitted.

There is way more than enough wacky stuff introduced by Unicode. Having dozens of letters A, for example. And giving a Japanese Kanji character the same code as a Chinese one that usually looks similar.

Yeah, that is quite inconsistent. Kanji literally means "Chinese Character" so it should be the same for the letter A. Unless a French A isn't equivalent to an English A.

Arabic numerals (0123456789) are not to be confused with the Arabic numerals (٠١٢٣٤٥٦٧٨٩). So the fact that kanji literally means "Chinese character" doesn't mean that kanji and hanzi should be considered the same script.

The Latin script (that which I write right now) and the Cyrillic script both derived heavily from the Greek script, especially the capital letters--fully 60% of them are identical in Latin and Greek, even more if you include obsolete letters like digamma and lunate sigma (roughly F and C, respectively). Most of these homoglyphs furthermore share identical phonetic values.

In retrospect, treating traditional Chinese, simplified Chinese, and Japanese kanji as different scripts seems like it would have been the better path. I don't know enough about the Korean and Vietnamese usage of Chinese characters to know if those scripts are themselves independent daughter scripts or complete imports of Chinese with a few extra things thrown in (consider Farsi's additions to Arabic, or Icelandic's þ and ð additions to Latin).

Re: Unicode Is Awesome

#73

Post by WalterGR »

ken wrote: post #24

> data to be transmitted in a byte, word or double word oriented format (i.e. in 8, 16 or 32-bits per code unit) I don't think I've heard "word" mean "16 bits" since the 1980's ... and apparently neither has Wikipedia: https://en.wikipedia.org/wiki/Word_(computer_architecture)#T...

The concept of the word size of an architecture is different than "word" which has long been used colloquially in computing to mean two bytes.

2 nibbles are a byte. 2 bytes are a word.

Edit: "Word" as two bytes may actually be a microcomputer-specific colloquialism.

Re: Unicode Is Awesome

#74

Post by mantap »

sansnomme wrote: post #63
ncmncm wrote: post #48

Earlier quoted context omitted.

There is way more than enough wacky stuff introduced by Unicode. Having dozens of letters A, for example. And giving a Japanese Kanji character the same code as a Chinese one that usually looks similar.

Yeah, that is quite inconsistent. Kanji literally means "Chinese Character" so it should be the same for the letter A. Unless a French A isn't equivalent to an English A.

The alternative is worse. Look at all of the problems we have with Turkish I, just because they didn't create new codepoints to make Turkish I and Latin I distinct even though they look the same.

Re: Unicode Is Awesome

#76

Post by euske »

jcranmer wrote: post #10
euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accuratel…

Unicode doesn't solve the underlying complexity of human languages, as you noted. I think the main contribution of Unicode Consortium is that they brought all the nitty-gritty problems of human languages into one central repository and made them visible to everyone. That itself is an awesome effort, and I personally had a lot of benefit from it (my native language is Japanese). But that doesn't make Unicode as a standard "awesome". Maybe we should be thankful for how messy it is? That's more or less a view that I can agree with.

Re: Unicode Is Awesome

#77

Post by kevin_thibedeau »

ncmncm wrote: post #65
mehrdadn wrote: post #53

Earlier quoted context omitted.

That might be wacky to you but I'm not sure it's wacky to the people to whom it makes a difference.

Lots of the wacky stuff came from the original dream of a purely 16-bit code, and then more wacky stuff to extend it from there. I.e., starting from UTF-8 could have avoided any amount of unpleasantness. But of course UTF-8 wasn't invented until later. The 16-bit representation got encrusted in OSes and languages of a certain period. The same goes, of course, for writing systems, going back to the first, that we woul…

Indiana doesn't have special time zones any more. It allows individual counties to choose which standard zone to be in but they all observe the normal DST.

Re: Unicode Is Awesome

#78

Post by kps »

jcranmer wrote: post #67
kps wrote: post #61

Earlier quoted context omitted.

‘Remove’ is too strong, since Unicode is entrenched. But there are things that should have been done differently. For instance, combining characters and operators should have been placed before the base character rather than after, so that (a) it would be possible to know when you've reached the end of a character^W glyph^W grapheme cluster without reading ahead, and (b) dead keys would be identical to the correspond…

> ASCII (1967) allowed for them: c BS , or , BS c ↦ ç and e BS ' or ' BS e ↦ é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 2C. Doesn't work for ñ, since the ASCII ~ is often typeset in the middle of the box instead of in a position to appear above an 'n' character. " is a pretty poor substitute for ◌̈ though, especially when you're trying to write ï as in naïve. And then there's the æ o…

Tilde exists in ASCII because of its use as an accent. (In 1967 the non-diacritic interpretation was an overline.) The use in programming languages, and lowering to fit other mathematical operators, came later.

There was never any requirement that ‘n BS ~’ have the same appearance as ‘n’ overprinted with ‘~’, although terminals capable of making the distinction didn't appear until the 70s.

Precomposed characters aren't relevant to illustrating composition mechanisms.

Re: Unicode Is Awesome

#79

Post by bawolff »

jancsika wrote: post #36
jcranmer wrote: post #10

Earlier quoted context omitted.

If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accuratel…

> such as façade and résumé That's simple: just url encode. Compare: www.façebook.com to www.fa%C3%A7ebook.com The second one is way easier to comprehend than the first.

You mean www.xn--faebook-vxa.com of course :P

Re: Unicode Is Awesome

#80

Post by jcranmer »

TheDong wrote: post #71
jcranmer wrote: post #59

Earlier quoted context omitted.

There is no character in EUC-JP or Shift-JIS that is not in Unicode--the explicit goal of Unicode in its original formulation was to be able to losslessly round-trip any other charset through Unicode, and the initial version of Unicode incorporated the source kanji lists for the EUC-JP/Shift-JIS charsets in their entirety.

That's true, but you misunderstood what I meant. The parent comment seemed to be implying that we should drop support for non-utf8 charsets. To me, that rings like saying a website with 'charset=EUC-JP' (such as http://www.os2.jp/ ) should be broken, as in browsers should error out or display a large quantity of black boxes due to it using a non-utf-8 encoding. I'm claiming the only reason the author thinks that's re…

Windows-1252/ISO-8859-1 (the two charsets are so commonly conflated that it's often best to treat them as one) was the dominant [non-ASCII] charset of the web until around 2007 or 2008, and their prevalence more recently is only about 5%.

A collection of Usenet messages gathered in 2014 (see http://quetzalcoatal.blogspot.com/2014/03/understanding-emai... for full details) showed that out of 1,000,000 messages, about 530,000 were actually ASCII; 270,000 were ISO-8859-1 or Windows-1252; and only 75,000 were UTF-8. More modern numbers would probably show higher UTF-8 counts, although Usenet is notoriously conservative in terms of technology.

What I'm trying to elucidate here is that the rise of UTF-8 isn't because most text is ASCII, but because there's been a rather more concerted effort to default content generation to UTF-8 and treat other charsets only as legacy inputs. Well, with the exception of the Japanese, who tend to be strongly averse to UTF-8. (I've been told that Japanese email users would rather have their text get silently mangled than silently converted to UTF-8 because you're quoting an email with a smart quote [not present in any of the 3 Japanese charsets], whereas every other locale was happy changing the default charset for writing to UTF-8).

Post reply on HN