Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emoji.length == 2

blog.jonnew.com

Post reply on HN
Print view View original

11–20 of 166 posts

Re: Emoji.length == 2

#11

Post by darkengine »

melloclello wrote: post #9
darkengine wrote: post #8

The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. To use the author's example: ‍woman - 1 codepoint black woman - 2 codepoints, w…

Man, imagine if you could compose chinese characters out of radicals like this. I'm not sure if that would be a good thing or a bad thing.

Somebody involved with Unicode must have had the same idea, because the ideographic description characters exist. However, I've never seen them used in practice because they don't actually render the character. You just get something like ⿰扌足, which corresponds to 捉.

https://en.wikipedia.org/wiki/Ideographic_Description_Charac...

Re: Emoji.length == 2

#12

Post by openasocket »

The issue doesn't really seem to be the emojis, but rather the variation sequences, which seem to be really awkward to work with, but I can sort of see why they're necessary. But the fact that we need special libraries to answer fairly basic queries about unicode text doesn't bode well.

Re: Emoji.length == 2

#13

Post by duskwuff »

melloclello wrote: post #9
darkengine wrote: post #8

The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. To use the author's example: ‍woman - 1 codepoint black woman - 2 codepoints, w…

Man, imagine if you could compose chinese characters out of radicals like this. I'm not sure if that would be a good thing or a bad thing.

The one that still surprises me is Hangul (Korean script). Hangul characters are made of 24 basic characters (jamo) which represent consonant and vowel sounds, which are composed into Hangul characters representing syllables.

Unicode has a block for Hangul jamo, but they aren't used in typical text. Instead, Hangul are presented using a massive 11K-codepoint block of every possible precomposed syllable. ¯\_(ツ)_/¯

Re: Emoji.length == 2

#14

Post by raphlinus »

TAForObvReasons wrote: post #6
phkahler wrote: post #2

Unicode is fucked. All these bullshit emojis remind me of the 1980s when ASCII was 7 bits but every computer manufacturer (Atari, Commodore, Apple, IBM, TI, etc...) made their own set of characters for the 128 values of a byte beyond ASCII. Of course Unicode is a global standard so your pile-of-poop emoji will still be a pile-of-poop on every device even if the amount of steam is different for some people. It's beyon…

There is a Unicode encoding "UTF-32" which has the advantage of being fixed width. This is not popular for the obvious reason that even ascii characters are expanded to 4 bytes. Additionally the windows APIs, among other interfaces, are not equipped to handle 4-byte codepages.

It's fixed width with respect to code points, but not with respect to any of the other things mentioned in the linked article. For example, the black heart with emoji variation selector (which makes it render red) is two code points.

Re: Emoji.length == 2

#15

Post by microtherion »

darkengine wrote: post #8

The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. To use the author's example: ‍woman - 1 codepoint black woman - 2 codepoints, w…

> It's like composing Mayan pictographs

Which reminds me: We need a Jaguar emoji!

Re: Emoji.length == 2

#16

Post by newtang »

danbruc wrote: post #5

The Unicode standard describes in Annex 29 [1] how to properly split strings into grapheme clusters. And here [2] is a JavaScript implementation. This is a solved problem. [1] http://www.unicode.org/reports/tr29/ [2] https://github.com/orling/grapheme-splitter

I'm the author. Thank you for you sharing! I will check it out shortly.

Re: Emoji.length == 2

#17

Post by darkengine »

duskwuff wrote: post #13
melloclello wrote: post #9

Earlier quoted context omitted.

Man, imagine if you could compose chinese characters out of radicals like this. I'm not sure if that would be a good thing or a bad thing.

The one that still surprises me is Hangul (Korean script). Hangul characters are made of 24 basic characters (jamo) which represent consonant and vowel sounds, which are composed into Hangul characters representing syllables. Unicode has a block for Hangul jamo, but they aren't used in typical text. Instead, Hangul are presented using a massive 11K-codepoint block of every possible precomposed syllable. ¯\_(ツ)_/¯

I would imagine this is a legacy from the Good Old Days when every Asian locale had its own encoding. Unicode imported the Hangul block from ISO-2022-KR/Windows-949 (different encodings of the same charset), which has only Hangul syllables.

Re: Emoji.length == 2

#18

Post by masklinn »

phkahler wrote: post #2

Unicode is fucked. All these bullshit emojis remind me of the 1980s when ASCII was 7 bits but every computer manufacturer (Atari, Commodore, Apple, IBM, TI, etc...) made their own set of characters for the 128 values of a byte beyond ASCII. Of course Unicode is a global standard so your pile-of-poop emoji will still be a pile-of-poop on every device even if the amount of steam is different for some people. It's beyon…

> Unicode is fucked. All these bullshit emojis

Ah yes, all those bloody emoji taking the place of better worthier characters, those dastardly pictures taking up all of one half of one 16th of one Unicode plane (which has only 16 of those, and only 14 public).

And the gall they have, actually being used and lighting up their section of plane 1 like a christmas tree while the rest of the plane lies in the darkness: http://reedbeta.com/blog/programmers-intro-to-unicode/heatma... what a disgrace, not only existing but being found useful, what has the world come to.

And then of course there's the technical side of things: emoji actually forced western developers — and especially anglo ones — to stop fucking up non-ASCII let alone non-BMP codepoints. I don't think it's a coincidence that MySQL finally added support for astral characters once emoji started getting prominent.

In fact, I have a pet theory that the rash of combining emoji in the latest revisions is in part a vehicle to teach developers to finally stop fucking up text segmentation and stop assuming every codepoint is a grapheme cluster.

Re: Emoji.length == 2

#19

Post by lacampbell »

melloclello wrote: post #9
darkengine wrote: post #8

The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. To use the author's example: ‍woman - 1 codepoint black woman - 2 codepoints, w…

Man, imagine if you could compose chinese characters out of radicals like this. I'm not sure if that would be a good thing or a bad thing.

https://en.wikipedia.org/wiki/Cangjie_input_method

This isn't at the textual level, and the components are not strictly radicals, but this may interest you.

Re: Emoji.length == 2

#20

Post by masklinn »

openasocket wrote: post #12

The issue doesn't really seem to be the emojis, but rather the variation sequences, which seem to be really awkward to work with, but I can sort of see why they're necessary. But the fact that we need special libraries to answer fairly basic queries about unicode text doesn't bode well.

> But the fact that we need special libraries to answer fairly basic queries about unicode text doesn't bode well.

That's always been needed to actually properly work with unicode, what do you think ICU is? Few if any languages have complete native Unicode support. And it's hardly new, Unicode has an annex (#29) dedicated to text segmentation: http://www.unicode.org/reports/tr29/

Post reply on HN