Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

151–159 of 159 posts

Re: Unicode Is Awesome

#151

Earlier quoted context omitted.

There actually is a rarely used, but standardized because it needed to be, UTF-16 variant UTF-EBCDIC, if you needed nightmares about it. [0] In some small decisions EBCDIC makes as much or more sense than ASCII; the decades of problems have been that ASCII and EBCDIC coexisted from basically the beginning. (IBM could have delayed the System/360 while ASCII was standardized and likely have saved decades of programmer…

> Maybe if IBM had something like UTF-8 as far back as the 1930s, AT&T needing backward compatibility with their teleprinters might not have been as big of a deal I think this points to why the science fiction scenario really is a science fiction scenario -- I think decoding and interpreting UTF-8, using it to control, say, a teleprompter, is probably significantly enough more expensive than ASCII that it would have…

Yeah, and I was thinking about it this morning the reliance of FF-FD for surrogate pairs would shred punch cards (too many holes) and probably be a big reason for IBM to dismiss it when they were hugely punch card dependent and hadn't made advances like the smaller square hole punches that could pack holes more densely and with better surrounding integrity.

"Sigh, another swiss cheese card jammed the reader from all these emojis."

Re: Unicode Is Awesome

#152

Earlier quoted context omitted.

There actually is a rarely used, but standardized because it needed to be, UTF-16 variant UTF-EBCDIC, if you needed nightmares about it. [0] In some small decisions EBCDIC makes as much or more sense than ASCII; the decades of problems have been that ASCII and EBCDIC coexisted from basically the beginning. (IBM could have delayed the System/360 while ASCII was standardized and likely have saved decades of programmer…

Yeah, no thanks to that nightmare, or to UTF-5. Did you know that UTF-7 is not a joke??

Yeah, email is one of those things that in a sci-fi scenario should probably be invented after UTF-8, and hopefully too with a better plan for binary attachments. uuencode/uudecode, MIME, base64, UTF-7, so many fun hacks on top of each other on top of a communications system restricted to ASCII to play things safe.

Re: Unicode Is Awesome

#153
post #148

Earlier quoted context omitted.

I don't see anything special in the linked image. The As look as they would in Latin script. (probably not what you meant, but just in case: the fourth letter is not a Cyrillic A but a D.)

I should have linked the English title as well, now I read my comment and see it is not sufficiently unambiguous. Here: https://en.wikipedia.org/wiki/File:The_Mandalorian_logo.jpg you can see that the A letters are rendered the same as Cyrillic Л. To further the confusion, in Buenos Aires there are many street signs that use Л as the letter A, and they use П as the letter N.

Part of the confusion was because I see the English title's As are actually Λ (capital Greek lambda), rather than Л (at least in the font that HN uses). I'm guessing from context that Л is sometimes rendered as Λ in some Cyrillic fonts.

Re: Unicode Is Awesome

#154

Earlier quoted context omitted.

> But now consider Hangul, where the combinations are to form syllables that are themselves glyphs... the problems get much worse now. What might your answer be to that? I don't have an answer, mostly because I know nothing about Hangul. Maybe decomposition is the right solution there. Frankly I don't care what Unicode does to solve the problems Hangul creates, and as Korea about 1% of the world's population I doubt…

> > it did not happen because of "egos" on the Unicode Consortium, or because of "national egos", or politics, or anything. > Did someone say that? Yes, u/kazinator did. > Anyway, it's pretty obvious why it happened. When a person invents new hammer, the second thing they do is going looking for all the other problems it might solve. That's not why decomposition happened. It happened because a) decomposition already…

> As to the idea that programmers who want nothing to do with I18N are the most common user or Unicode, that's rather insulting to the real users: the end users.

Oh for Pete's sake. Unicode / ASCII / ISO 8859-1 are encoding computers and thus programmers use to represent text. Users don't read Unicode, they read text. They never, ever have to deal with Unicode and most wouldn't know what it was if it leapt up and hit them in the faxe, so if Unicode justified adding features to accommodate these non-existent users, I guess that explains how we got into this mess.

Re: Unicode Is Awesome

#155

Earlier quoted context omitted.

Is there a realitvely easy way to find the character set that was included for uppercase superscript W? (ᵂ)

In the case of U+1D42 Modifier Letter Capital W I was wrong about the cause, it was in fact specifically added on the rationale that for this purpose (phonetics) the presentation was semantic in nature, and so the plain text (thus Unicode) needed to preserve these symbols which could otherwise be handled by a presentation layer. U+1D42 Modifier Letter Capital W was added in Unicode 4.0 as part of the Phonetic Extensi…

Does that mean there is hope to complete the [a-Z] sub/super set?

Re: Unicode Is Awesome

#156

Earlier quoted context omitted.

> > it did not happen because of "egos" on the Unicode Consortium, or because of "national egos", or politics, or anything. > Did someone say that? Yes, u/kazinator did. > Anyway, it's pretty obvious why it happened. When a person invents new hammer, the second thing they do is going looking for all the other problems it might solve. That's not why decomposition happened. It happened because a) decomposition already…

> As to the idea that programmers who want nothing to do with I18N are the most common user or Unicode, that's rather insulting to the real users: the end users. Oh for Pete's sake. Unicode / ASCII / ISO 8859-1 are encoding computers and thus programmers use to represent text. Users don't read Unicode, they read text. They never, ever have to deal with Unicode and most wouldn't know what it was if it leapt up and hit…

They read text in many scripts (because many writers use more than one script, and many readers can read more than one script). Without Unicode you can usually use TWO scripts: ASCII English + one other (e.g., ISO8859-*, SHIFT_JIS, ...). There are NO OTHER CODESETS than Unicode that can support THREE or more scripts, or any combination of TWO where one isn't ASCII English. For example, Indian subcontinent users are very likely to speak multiple languages and use at least two scripts other than ASCII English. Besides normal people all over the world who have to deal with multiple scripts, there's also: scolars, diplomats, support staff at many multi-nationals, and many others who also need to deal with multiple scripts.

Whether you like it or not, Unicode exists to make USERS' lives better. Programmers?? Pfft. We can deal with the complexity of all of that. User needs, on the other hand, simply cannot be met reasonably with any alternatives to Unicode.

Re: Unicode Is Awesome

#157
post #148

Earlier quoted context omitted.

I should have linked the English title as well, now I read my comment and see it is not sufficiently unambiguous. Here: https://en.wikipedia.org/wiki/File:The_Mandalorian_logo.jpg you can see that the A letters are rendered the same as Cyrillic Л. To further the confusion, in Buenos Aires there are many street signs that use Л as the letter A, and they use П as the letter N.

Part of the confusion was because I see the English title's As are actually Λ (capital Greek lambda), rather than Л (at least in the font that HN uses). I'm guessing from context that Л is sometimes rendered as Λ in some Cyrillic fonts.

> I'm guessing from context that Л is sometimes rendered as Λ in some Cyrillic fonts.

Exactly. It is more often rendered like that in Bulgaria. But it is still the letter Л.

Which just furthers the point that glyph rendering and character code points are very different problems and the multiple code points in Unicode are the right approach.

Re: Unicode Is Awesome

#158

Earlier quoted context omitted.

> As to the idea that programmers who want nothing to do with I18N are the most common user or Unicode, that's rather insulting to the real users: the end users. Oh for Pete's sake. Unicode / ASCII / ISO 8859-1 are encoding computers and thus programmers use to represent text. Users don't read Unicode, they read text. They never, ever have to deal with Unicode and most wouldn't know what it was if it leapt up and hit…

They read text in many scripts (because many writers use more than one script, and many readers can read more than one script). Without Unicode you can usually use TWO scripts: ASCII English + one other (e.g., ISO8859-*, SHIFT_JIS, ...). There are NO OTHER CODESETS than Unicode that can support THREE or more scripts, or any combination of TWO where one isn't ASCII English. For example, Indian subcontinent users are v…

> Whether you like it or not, Unicode exists to make USERS' lives better.

They've been using multiple scripts long before computers. When computers came along those users quite reasonably demanded they be able to write the same scripts. This created a problem for the programmers. The obvious solution is a universal set of code points - and ISO 10646 was born. It was not rocket science. But if it had not come along some other hack / kludge would have been used because the market is too large to be abandoned by the computer companies. They would have put us programmers in a special kind of hell, but I can guarantee the users would not have known about that, let alone cared.

Oddly the encoding schemes proposed by ISO 10646 universally sucked. Unicode walked into that vacuum with their first cock up - 16 bits was enough for anybody. It was not just dumb because it was wrong - it was dumb because they didn't propose unique encoding. They gave us BOM markers instead. Double fail. They didn't win because the one thing they added, their UCS-2 encoding, was any better than what came before. They somehow managed to turn it into a Europe vs USA popularity contest. Nicely played.

Then Unicode and 10646 became the same thing. They jointly continued on in the same manner as before, inventing new encodings to paper over the UCS-2 mistake. Those new encodings all universally sucked. The encoding we programmers use today, UTF-8, was invented by, surprise, surprise, a programmer who was outside of Unicode / 10646 group think. It was popularised at a programmers conference, USENIX, and from there on it was obvious was going to be used regardless of what Unicode / 10646 thought of it, so they accepted it.

If Perl6 is any indication, the programmers are getting the shits with the current mess. Perl6 has explicitly added functions that treat text as a stream of grapheme's rather than Unicode code points. The two are the same of course except when your god damned compositions rear their ugly head - all they do is eliminate that mess. Maybe you should take note. Some programmers have already started using something other than Unicode because it makes their jobs easier. If it catches on you are going to find out real quickly just how much the users care about the encoding scheme computers use for text.

None of this is to trivialise the task of assigning code points to grapheme's. It's huge task. But for pete's sake don't over inflate your ego's by claiming you are doing some great service for mankind. The only thing on the planet that uses the numbers Unicode assigns to characters as is computers. Computers are programmed by one profession - programmers. Your entire output is consumed by that one profession. Yet for all the world what you've written here seems to say Unicode has some higher purpose, and you are optimising for that, whatever it may be. For gods same come down to earth.

Re: Unicode Is Awesome

#159

Earlier quoted context omitted.

They read text in many scripts (because many writers use more than one script, and many readers can read more than one script). Without Unicode you can usually use TWO scripts: ASCII English + one other (e.g., ISO8859-*, SHIFT_JIS, ...). There are NO OTHER CODESETS than Unicode that can support THREE or more scripts, or any combination of TWO where one isn't ASCII English. For example, Indian subcontinent users are v…

> Whether you like it or not, Unicode exists to make USERS' lives better. They've been using multiple scripts long before computers. When computers came along those users quite reasonably demanded they be able to write the same scripts. This created a problem for the programmers. The obvious solution is a universal set of code points - and ISO 10646 was born. It was not rocket science. But if it had not come along so…

Thank you for your reference to Perl 6. Please note that Perl 6 has been renamed to "Raku" (https://raku.org) using #rakulang as a tag for social media.

Please also not that all string handling in Raku (the `Str` class) is based on graphemes, not just some added functions. This e.g. means that a newline is always 1 grapheme (regardless of whether it was a CR or LF or CRLF). And that for é there is no difference between é (LATIN SMALL LETTER E WITH ACUTE aka 0x00E9) and (LATIN SMALL LETTER E, COMBINING ACUTE ACCENT aka 0x0065, 0x0301).

Please note that for any combination between characters and combiners for which there does not exist a composed version in Unicode, Raku will create synthetic codepoints on the fly. This ensures that you can consider your texts as graphemes, but still be able to roundtrip strange combinations.

Post reply on HN