Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

Post reply on HN
Print view View original

61–70 of 159 posts

Re: Unicode Is Awesome

#61

Post by kps »

jcranmer wrote: post #10
euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accuratel…

‘Remove’ is too strong, since Unicode is entrenched. But there are things that should have been done differently. For instance, combining characters and operators should have been placed before the base character rather than after, so that (a) it would be possible to know when you've reached the end of a character^W glyph^W grapheme cluster without reading ahead, and (b) dead keys would be identical to the corresponding characters.

> façade and résumé

ASCII (1967) allowed for them: c BS , or , BS c ↦ ç and e BS ' or ' BS e ↦ é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 2C.

Re: Unicode Is Awesome

#62

Post by kazinator »

jrochkind1 wrote: post #60
kazinator wrote: post #52

Earlier quoted context omitted.

> As a developer who's been working intimately with user-facing strings for years, ... User-facing is easy; things go downhill when users have system-facing strings of their own, and some of those strings become other-user-facing strings. > with only an extremely few things I think might have served better if done differently. Thus, in spite of disagreeing in the strongest possible terms, you do have some nits to pic…

Your idea that either there are NO "nits to pick" (things that could have been done better in a standard, complete perfection), OR it means that the standards-makers "decided to make it complicated to bolster their egos" -- is ABSOLUTELY INSANE.

My point isn't that there must be no nits to pick, but that look, even a self-proclaimed Unicode cheerleader who disagrees with me in the "strongest possible" terms still finds it necessary to mention that he or she has some.

Re: Unicode Is Awesome

#63

Post by sansnomme »

ncmncm wrote: post #48
jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

There is way more than enough wacky stuff introduced by Unicode. Having dozens of letters A, for example. And giving a Japanese Kanji character the same code as a Chinese one that usually looks similar.

Yeah, that is quite inconsistent. Kanji literally means "Chinese Character" so it should be the same for the letter A. Unless a French A isn't equivalent to an English A.

Re: Unicode Is Awesome

#64

Post by stefco_ »

zzo38computer wrote: post #32
magicalhippo wrote: post #26

Earlier quoted context omitted.

> You can't effectively use one character set well for everything; different applications have different requirements. In our application, our users gets data from systems around the world, and might have to change some of it before sending a file with the data to some official system. The data includes names of people and places. How would you do this using character sets? One file might need to contain names with C…

> Or, just maybe, strings in the file could be Unicode, encoded in say UTF-8, so that the handling of all of them are uniform... Actually, that won't work. There are cases where a character may be different according to the language, where capitalization may differ depending on the language, where sort order may depend on the language, etc.

If your application is allowing users to edit the text, or if you know which languages will be used, or if you don't care about capitalization, then you don't have to worry about any of those edge cases, and Unicode is useful.

Re: Unicode Is Awesome

#65

Post by ncmncm »

mehrdadn wrote: post #53
ncmncm wrote: post #48

Earlier quoted context omitted.

There is way more than enough wacky stuff introduced by Unicode. Having dozens of letters A, for example. And giving a Japanese Kanji character the same code as a Chinese one that usually looks similar.

That might be wacky to you but I'm not sure it's wacky to the people to whom it makes a difference.

Lots of the wacky stuff came from the original dream of a purely 16-bit code, and then more wacky stuff to extend it from there. I.e., starting from UTF-8 could have avoided any amount of unpleasantness. But of course UTF-8 wasn't invented until later. The 16-bit representation got encrusted in OSes and languages of a certain period.

The same goes, of course, for writing systems, going back to the first, that we would all do differently in hindsight.

Even today we are making apparently sensible choices we (or our digital successors) will regret as deeply.

ISO 8601 looks good now, but it only delays the transition to a rational calendar which, admittedly, we would certainly get wrong if we tried codifying one today.

Fortunately daylight saving time will be gone worldwide before the next decade passes, but not without some places getting stuck at the wrong timezone. (E.g. Portugal different from Spain, and probably Indiana different from itself.)

Re: Unicode Is Awesome

#66

Post by gpvos »

flohofwoe wrote: post #21
wvenable wrote: post #17

Earlier quoted context omitted.

Windows and many other operating systems and languages (Java) got on board with Unicode back when the character set would fit in 16bits. The character set originally used was UCS-2 (not UTF-16). UTF-16 came next to extend the Unicode character set beyond 65536 code points. UTF-8 wasn't even invented until well after all these operating systems and languages deployed Unicode. They didn't see the light of day to use UT…

I actually checked a while ago when UTF-8 was created, and it was just around the same time when Windows NT was developed with 16-bit "early" Unicode support. UTF-8 was created in September 1992 [1], and Windows NT came out mid 1993, but I guess it was too late for Windows to change to UTF-8 (and I guess the advantages of UTF-8 haven't been as clear back then). But IMHO there's no excuse to not use UTF-8 after around…

Also, UTF-16 was only published in July 1996 (although the need for more than 16 bits was probably apparent a bit earlier). So before that, Unicode was only a 16-bit encoding, and UCS-2 was enough. UTF-8 was initially just a nice trick to keep using ASCII characters for things like directory separators (/) and single-byte NUL terminators. By 1995 its superiority certainly wasn't apparent yet.

Also, Windows internals were completely 16-bit-character based, including e.g. the NTFS disk format, so by 1992 that was already quite hard to change.

That said, it is crazy that NT didn't have full UTF-8 support, including in console windows, by about 2000.

Re: Unicode Is Awesome

#67

Post by jcranmer »

kps wrote: post #61
jcranmer wrote: post #10

Earlier quoted context omitted.

If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accuratel…

‘Remove’ is too strong, since Unicode is entrenched. But there are things that should have been done differently. For instance, combining characters and operators should have been placed before the base character rather than after, so that (a) it would be possible to know when you've reached the end of a character^W glyph^W grapheme cluster without reading ahead, and (b) dead keys would be identical to the correspond…

> ASCII (1967) allowed for them: c BS , or , BS c ↦ ç and e BS ' or ' BS e ↦ é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 2C.

Doesn't work for ñ, since the ASCII ~ is often typeset in the middle of the box instead of in a position to appear above an 'n' character. " is a pretty poor substitute for ◌̈ though, especially when you're trying to write ï as in naïve. And then there's the æ of archæology, which doesn't work with overwriting.

I'll also point out that ç is U+00E7 in Unicode and C3 A7 in UTF-8, not 63 CC A6, since it's a precomposed character (and NFC form is usually understood to be the preferred way to normalize Unicode unless there's a reason to do something else).

Re: Unicode Is Awesome

#68

Post by bluetwo »

euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

Without arguing the details, I have to agree with your statement because the article never really supported it's claim of being awesome.

Re: Unicode Is Awesome

#69

Post by Shorel »

sansnomme wrote: post #63
ncmncm wrote: post #48

Earlier quoted context omitted.

There is way more than enough wacky stuff introduced by Unicode. Having dozens of letters A, for example. And giving a Japanese Kanji character the same code as a Chinese one that usually looks similar.

Yeah, that is quite inconsistent. Kanji literally means "Chinese Character" so it should be the same for the letter A. Unless a French A isn't equivalent to an English A.

Cyrillic А is not the same as English A.

For example, some fonts render A in a way that looks like Cyrilic Л. (Like The Mandalorian title screen.)

This would be incorrect if using the same A for both: https://i.ytimg.com/vi/V8fC7bdV-mI/maxresdefault.jpg

Re: Unicode Is Awesome

#70

Post by pdonis »

userbinator wrote: post #40
jcranmer wrote: post #10

Earlier quoted context omitted.

If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accuratel…

If you extend ASCII to CP1252, which is the most common encoding besides/before UTF-8 became common, then you do get those accented characters (and that's likely responsible for the popularity of '1252.) In fact, the first 256 characters of Unicode are almost identical to CP1252. I'm pretty sure that's not a coincidence.

> the first 256 characters of Unicode are almost identical to CP1252. I'm pretty sure that's not a coincidence.

That depends on whether you consider the fact that Windows CP 1252 is almost identical to Latin-1 (ISO-8859-1), which is exactly the first 256 characters of Unicode, to be a coincidence.

Post reply on HN