Just posted another Unicode article to HN.
"Hacking GitHub with Unicode"
Unicode Is Awesome
131–140 of 159 posts
Re: Unicode Is Awesome
#132Unicode definitely has flaws but that doesn't mean we should throw the baby out with the bathwater and go back to "ASCII and other character sets." There's a reason we moved on from that world. However, I bet we will see another encoding coming up eventually (within 30 years) which solves the problems Unicode currently has and introduces a new set of problems as well. I saw this comment [0] about how that encoding sh…
Unicode URL has serious security problems. The canonical example is google.com vs gооgle.com.
Commented above, but to follow up from yesterday, here is the next post.
"Hacking GitHub with Unicode" https://news.ycombinator.com/item?id=21693550
Re: Unicode Is Awesome
#133Earlier quoted context omitted.
The proposed UCS-G-8 encoding [1] does exactly that. And in case UTF-16 never dies, the website also proposes extensions to UTF-16 (and UTF-32) as well. [1] http://ucsx.org/g8
As a lay-developer, I know unicode is what you need for international character support. Oh, so what are the options: utf-8, utf-16, utf-32. I will choose utf-32 just because 32 > 16 or 8.
I'm not sure if that's a joke (maybe that's why it got downvoted).
But the answer is that 32 is not better than 16, which is not better than 8, in this specific case. The bit count here is about memory efficiency. There are developers who think/thought that UTF-32 would improve random access into strings because it's a fixed-sized encoding of codepoints, but in fact it does not because there are glyphs that require multiple codepoints.
All who pass must abandon the idea of random access into Unicode strings!
Once you give up on Unicode string random access, you can focus on other reasons to like one UTF over another, and then you quickly realize that UTF-8 is the best by far.
For example there's this misconception that UTF-8 takes significantly (e.g., 50%) more space than UTF-16 when handling CJK, but that's not obviously true -- I've seen statistical analyses showing that UTF-8 is roughly comparable to UTF-16 in this regard.
Ultimately UTF-8 is much easier to retrofit into older systems that use ASCII, it's much easier to use in libraries and programs that were designed for ASCII, and doesn't have the fundamental 21-bit codespace limit that UTF-16 has.
Re: Unicode Is Awesome
#134Earlier quoted context omitted.
Unicode was originally designed to fit in 16 bits, and this is memorialized in Java APIs that make it easy to mess up. The unicode character does not specify the glyph to draw. Han unification is the best known, but not only source, of this challenge. The glyph does not specify the unicode character. Precombined vs combining characters is a source of this challenge. The result is that a name can be entered into a dat…
One of your points is that an encoding designed to handle languages has support for more than one kind of white space. Given that languages use more than one kind of white space, this is sort of a necessity. Another one is that a standard designed to support all languages has a feature necessary for supporting some languages. Those aren't inconsistencies, so do feel free to go on.
One of your points is that an encoding designed to handle languages has support for more than one kind of white space.
No. It is that there is more than one kind of invisible character. No language has invisible characters.
Another one is that a standard designed to support all languages has a feature necessary for supporting some languages.
Not sure what point you are misreading here. But that was not among my points.
Re: Unicode Is Awesome
#135Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…
If we could go back in time to Unicode's beginning and start over but with all that we know today... Unicode would still look a lot like what it looks like today, except that: - UTF-8 would have been specified first - we'd not have had UCS-2, nor UTF-16 - we'd have more than 21 bits of codespace - CJK unification would not have been attempted - we might or might not have pre-composed codepoints[0] - a few character-s…
> Or IBM in the 30s
Love an interesting sci-fi scenario. UTF-8 was a really neat technical trick, and a lot of the early UTF-8 technical documentation was already on IBM letterhead. I think if you showed up with the right documents at various points in history IBM would have been ecstatic to have an idea like UTF-8, at least. UTF-8 would have sidestepped a lot of mistakes with code pages and CCSIDs (IBM's attempts at 16-bit characters, encoding both code page and character), and IBM developers likely would have enjoyed that. Also, they might have been delightfully confused about how the memo was coming from inside the house by coworkers not currently on payroll.
Possibly that even extends as far back as the 1930s and formation of the company, because even then IBM aspired to be a truly International company, given the I in its own name.
I'm not sure how much of the rest of Unicode you could have convinced them of, but it's funny imagining explaining say Emoji to IBM suits at various points in history.
Re: Unicode Is Awesome
#136Earlier quoted context omitted.
If we could go back in time to Unicode's beginning and start over but with all that we know today... Unicode would still look a lot like what it looks like today, except that: - UTF-8 would have been specified first - we'd not have had UCS-2, nor UTF-16 - we'd have more than 21 bits of codespace - CJK unification would not have been attempted - we might or might not have pre-composed codepoints[0] - a few character-s…
> Or IBM in the 30s Love an interesting sci-fi scenario. UTF-8 was a really neat technical trick, and a lot of the early UTF-8 technical documentation was already on IBM letterhead. I think if you showed up with the right documents at various points in history IBM would have been ecstatic to have an idea like UTF-8, at least. UTF-8 would have sidestepped a lot of mistakes with code pages and CCSIDs (IBM's attempts at…
I.. agree. After all, even before ASCII people already used the English character set and punctuation to type out many non-English Latin characters on typewriters (did typesetters do that with movable type, ever? I dunno, but I imagine so). Just as ASCII was intended to work with overstrike for internationalization, i can imagine combining codepoints having been a thing even earlier.
OTOH, it wouldn't have been UTF-8 -- it would have been an EBCDIC-8 thing, and probably not good :)
Re: Unicode Is Awesome
#137Earlier quoted context omitted.
> UTF-8 isn't such a bad encoding (although it isn't ideal for fix pitch text; I invented a character set and encoding which would be better for fix pitch text). This is utterly incoherent.
Can anyone explain how the statement I responded to makes sense? I must be wrong, getting so many disagreements.
Well, UTF-8 is an encoding of Unicode, which allows for surrogate pairs and all that jazz which can be a bad fit for fixed-pitch text.
For example, take a Zalgo text generator[1] and try to make the result make sense in a fixed-pitch (monospace) setting.
At least that's my interpretation of what he tried to convey.
[1]: http://eeemo.net/
Re: Unicode Is Awesome
#138Earlier quoted context omitted.
One of your points is that an encoding designed to handle languages has support for more than one kind of white space. Given that languages use more than one kind of white space, this is sort of a necessity. Another one is that a standard designed to support all languages has a feature necessary for supporting some languages. Those aren't inconsistencies, so do feel free to go on.
One of your points is that an encoding designed to handle languages has support for more than one kind of white space. No. It is that there is more than one kind of invisible character. No language has invisible characters. Another one is that a standard designed to support all languages has a feature necessary for supporting some languages. Not sure what point you are misreading here. But that was not among my point…
You said "But there are whole languages (particularly from the Indian subcontinent) that cannot be written without combining characters."
I suppose I didn't consider that they could be written without combining characters given a different design.
As far as invisible characters, I'm not interested in arguing about it. English, as written, has all sorts of different structural uses of white space, it isn't all just style.
Re: Unicode Is Awesome
#139Earlier quoted context omitted.
One of your points is that an encoding designed to handle languages has support for more than one kind of white space. No. It is that there is more than one kind of invisible character. No language has invisible characters. Another one is that a standard designed to support all languages has a feature necessary for supporting some languages. Not sure what point you are misreading here. But that was not among my point…
You said "But there are whole languages (particularly from the Indian subcontinent) that cannot be written without combining characters." I suppose I didn't consider that they could be written without combining characters given a different design. As far as invisible characters, I'm not interested in arguing about it. English, as written, has all sorts of different structural uses of white space, it isn't all just st…
I suppose I didn't consider that they could be written without combining characters given a different design.
They could be.
Likewise European languages can be written without precombined characters. The fact that é can be written in multiple ways was my point.
As far as invisible characters, I'm not interested in arguing about it. English, as written, has all sorts of different structural uses of white space, it isn't all just style.
You still don't understand. I am not talking about whitespace. I am talking about invisible zero-width characters that can be slipped into text with no sign that they are there. Characters like U+180E, U+200B, U+FEFF, U+200C, U+200D, and U+FEFF. Not to mention that you can achieve the same thing with control characters like U+200FU+200E. (The undetectability of the last one is language dependent.)
As I said, this can be used to invisibly sign a document. But I don't see any other particular point to having so many ways to accomplish what looks like nothing.
Re: Unicode Is Awesome
#140Earlier quoted context omitted.
> Or IBM in the 30s Love an interesting sci-fi scenario. UTF-8 was a really neat technical trick, and a lot of the early UTF-8 technical documentation was already on IBM letterhead. I think if you showed up with the right documents at various points in history IBM would have been ecstatic to have an idea like UTF-8, at least. UTF-8 would have sidestepped a lot of mistakes with code pages and CCSIDs (IBM's attempts at…
I.. agree. After all, even before ASCII people already used the English character set and punctuation to type out many non-English Latin characters on typewriters (did typesetters do that with movable type, ever? I dunno, but I imagine so). Just as ASCII was intended to work with overstrike for internationalization, i can imagine combining codepoints having been a thing even earlier. OTOH, it wouldn't have been UTF-8…
There actually is a rarely used, but standardized because it needed to be, UTF-16 variant UTF-EBCDIC, if you needed nightmares about it. [0]
In some small decisions EBCDIC makes as much or more sense than ASCII; the decades of problems have been that ASCII and EBCDIC coexisted from basically the beginning. (IBM could have delayed the System/360 while ASCII was standardized and likely have saved decades of programmer grief.) The reasons that UTF-EBCDIC is so bad (such as that it is always a 16-bit encoding) could likely have been avoided had IBM awareness of UTF-8 ahead of time.
Maybe if IBM had something like UTF-8 as far back as the 1930s, AT&T needing backward compatibility with their teleprinters might not have been as big of a deal and ASCII might have been more IBM dominated. Or since this is a sci-fi scenario, you just impress on IBM that they need to build a telegraph compatible teleprinter model or three in addition to all their focus on punch cards, and maybe they'd get all of it interoperable themselves ahead of ASCII.
Though that starts to ask about the scenario what happens if you give UTF-8 to early Baudot code developers in the telegraph world. You might have a hard time to convince them they need more than 5-bits, but if you could accomplish that, imagine where telegraphy could have gone. Winking face emoji full stop