Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

Post reply on HN
Print view View original

21–30 of 159 posts

Re: Unicode Is Awesome

#21

Post by flohofwoe »

wvenable wrote: post #17
flohofwoe wrote: post #11

IMHO the article should mention that UTF-16 was (more or less) a hack to fix Windows and some other systems which didn't see the light and use UTF-8 from the start. UTF-16 has all the disadvantes of UTF-8 (variable length) and UTF-32 (endianess), but none of the advantages (encoding as endian-agnostic, 7-bit ASCII compatible byte stream like UTF-8, or a fixed-width encoding like UTF-32). UTF-16 should really be consi…

Windows and many other operating systems and languages (Java) got on board with Unicode back when the character set would fit in 16bits. The character set originally used was UCS-2 (not UTF-16). UTF-16 came next to extend the Unicode character set beyond 65536 code points. UTF-8 wasn't even invented until well after all these operating systems and languages deployed Unicode. They didn't see the light of day to use UT…

I actually checked a while ago when UTF-8 was created, and it was just around the same time when Windows NT was developed with 16-bit "early" Unicode support. UTF-8 was created in September 1992 [1], and Windows NT came out mid 1993, but I guess it was too late for Windows to change to UTF-8 (and I guess the advantages of UTF-8 haven't been as clear back then).

But IMHO there's no excuse to not use UTF-8 after around 1995 ;)

[1] https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt

Re: Unicode Is Awesome

#22

Post by deathanatos »

I swear there should be some rule or law about how Unicode articles will inevitably muddle code units/points / grapheme clusters / bytes together.

> String length is typically determined by counting codepoints.

> This means that surrogate pairs would count as two characters.

If you were counting code points, a surrogate pair would be 1. If it's two, you're counting code units.

> Combining multiple diacritics may be stacked over the same character. a + ̈ == ̈a, increasing length, while only producing a single character.

Not if you're counting code points or code units, which would both produce an answer of "2", and that's a great example of why you shouldn't count with either.

The dark blue on black in tables is next to invisible. And then to put that on white on the alternate rows is just eyeball murder.

> Since there are over 1.1 million UTF-8 glphys (sic)

UTF-8 glyphs twitch; aside from that, I'm really curious how they got that number. In some ways, a font has it easy; my understanding is that modern font formats can do one glyph for acute accent, one glyph for all the vowels/letters, and then compose the glyphs into arrangements for having them combined. (IDK if those are also "glyphs" to the font or not.) But it's less drawing, at least. OTOH, some characters have >1 appearance/"image", AIUI.

Re: Unicode Is Awesome

#23

Post by akersten »

Unicode is an inspirational standard. We started with so many different character encodings and wound up pretty universally using Unicode. I wouldn't be surprised to see browsers start to drop support for other encodings - who even uses them at this point?

Are there any scenarios where you wouldn't use Unicode, other than an every-byte-matters embedded system?

Re: Unicode Is Awesome

#25

Post by jontro »

ken wrote: post #24

> data to be transmitted in a byte, word or double word oriented format (i.e. in 8, 16 or 32-bits per code unit) I don't think I've heard "word" mean "16 bits" since the 1980's ... and apparently neither has Wikipedia: https://en.wikipedia.org/wiki/Word_(computer_architecture)#T...

In Windows world DWORD is 32 bits large: https://docs.microsoft.com/en-us/openspecs/windows_protocols...

Re: Unicode Is Awesome

#26

Post by magicalhippo »

zzo38computer wrote: post #12
euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

I think Unicode is terrible. Remove everything. Use ASCII and other character sets. Unicode is OK for searching for data using many different languages (if you omit much of the junk such as emoji and compatibility characters), although might not be best with that too. You can't effectively use one character set well for everything; different applications have different requirements. Unicode is equally bad for everyth…

> You can't effectively use one character set well for everything; different applications have different requirements.

In our application, our users gets data from systems around the world, and might have to change some of it before sending a file with the data to some official system. The data includes names of people and places. How would you do this using character sets?

One file might need to contain names with Cyrillic characters and with Norwegian characters. There's no character set with both. Should each string in the file have an attribute saying which character set the string is encoded in? What are the odds that people implementing that won't mess that up when oh so many can't even get a single encoding attribute right[1]?

Or, just maybe, strings in the file could be Unicode, encoded in say UTF-8, so that the handling of all of them are uniform...

[1]: https://www.w3.org/TR/xml/#charencoding

Re: Unicode Is Awesome

#27

Post by flohofwoe »

akersten wrote: post #23

Unicode is an inspirational standard. We started with so many different character encodings and wound up pretty universally using Unicode. I wouldn't be surprised to see browsers start to drop support for other encodings - who even uses them at this point? Are there any scenarios where you wouldn't use Unicode, other than an every-byte-matters embedded system?

Sorry for nitpicking but: Unicode is not an encoding, just (basically) a central registry for numbers, and you can have Unicode strings made of bytes (the UTF-8 encoding), in fact that's the most useful encoding for exchanging text data :)

Re: Unicode Is Awesome

#28

Post by grantmnz »

If you'd like to explore Unicode characters, you can use the Unicode Character finder, a web app I built some years ago: https://www.mclean.net.nz/ucf/

The app allows you to paste in a character to find out more about it, or to search the database of character descriptions to find what you're after.

You can link to a specific character to share with your friends and family: https://www.mclean.net.nz/ucf/?c=U+130BA

Re: Unicode Is Awesome

#29

Post by squaresmile »

Unicode definitely has flaws but that doesn't mean we should throw the baby out with the bathwater and go back to "ASCII and other character sets." There's a reason we moved on from that world. However, I bet we will see another encoding coming up eventually (within 30 years) which solves the problems Unicode currently has and introduces a new set of problems as well. I saw this comment [0] about how that encoding should get started.

> Greek, for example, has a lot of special-casing in Unicode. Korean is devilishly hard to render correctly the way Unicode handles it. And once you get into the right-to-left scripts, scripts that sort-of-sometimes omit vowels, or Devanagari (the script used to write a bunch of widely-spoken languages in India), you start needing very different capabilities than what's involved in Western European writing. _The better approach probably would have been to start with those, and work back to the European scripts_

[0] https://www.reddit.com/r/programming/comments/b09c0j/when_zo...

Funnily enough, URLs still can't do actual Unicode.

Post reply on HN