Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emoji.length == 2

blog.jonnew.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: Emoji.length == 2

#61

Post by pcwalton »

Animats wrote: post #47
TazeTSchnitzel wrote: post #38

Earlier quoted context omitted.

> We got into this because some people thought that single-color emoji were racist. Emoji skin colour and emoji using colour are two unrelated concerns.

Not if you have to render them, or store them in a font. There are now color fonts.[1] Firefox and Microsoft Edge now support them. It's done with SVG artwork inside OpenType font files. [1] https://color.typekit.com/

There are multiple ways to encode colors in OpenType fonts: not only SVG (which is probably the least common way), but also the Microsoft COLR table and PNG raster glyphs (used by Apple).

Re: Emoji.length == 2

#62

Post by Manishearth »

darkengine wrote: post #52
Manishearth wrote: post #41

Earlier quoted context omitted.

> The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. No, no, no, NO. Please stop spreading this bit of misinformation. Emoji has n…

That is why I said "with moderate success". It's not 100% reliable, but mostly good enough for basic cases. Twitter, for example, used to do an NFC normalization and count codepoints to enforce the 140 charcater limit, and this was close enough to or exactly right for enforcing 140 graphemes in probably 98% of text on Twitter. They can no longer do this because some new emoji can consume 7 codepoints for a single gra…

No, it's not, it's "mostly good enough for some scripts". This attitude just ends up making software that sucks for a subset of scripts.

Twitter is not a "basic case", it's a case where the length limit is arbitrary and the specifics don't matter much anymore. Usually when you want to segment text that is not the case.

Edit: basically, my problem with your original comment is that it helps spread the misinformation that a lot of these things are "just" emoji problems, so some folks tend to ignore them since they don't care about emoji that much, and real human languages suffer.

Re: Emoji.length == 2

#63

Post by masklinn »

Animats wrote: post #47
TazeTSchnitzel wrote: post #38

Earlier quoted context omitted.

> We got into this because some people thought that single-color emoji were racist. Emoji skin colour and emoji using colour are two unrelated concerns.

Not if you have to render them, or store them in a font. There are now color fonts.[1] Firefox and Microsoft Edge now support them. It's done with SVG artwork inside OpenType font files. [1] https://color.typekit.com/

> Not if you have to render them, or store them in a font.

I think you misunderstood the comment to which you replied. What it was saying is that emoji fitzpatrick scale and color fonts are orthogonal concerns, you can do skin tones in grayscale.

Re: Emoji.length == 2

#65

Post by TorKlingberg »

If you want to do Unicode correctly, you shouldn't ask for the "length" of a string. The is no true definition of length. If want to know how many bytes it uses in storage, ask for that. If you want to know how wide it will be on the screen, ask for that. Do not iterate over strings character by character.

Re: Emoji.length == 2

#66

Post by toast0 »

kmill wrote: post #35

There are multiple ways of counting "length" of a string. Number of UTF-8 bytes, number of UTF-16 code units, number of codepoints, number of grapheme clusters. These are all distinct yet valid concepts of "length." For the purpose of allocating buffers, I can see the obvious use in knowing number of bytes, UTF-16 code units, or the number of codepoints. I also see the use in being able to iterate through grapheme cl…

> Perhaps someone can shed light on a compelling use case for knowing the number of grapheme clusters in a particular string, because I haven't been able to think of one.

If you have a limit on the length of a field, it helps to tell the user what it is in a way they understand. For non-technical users, bytes (and the embedded issue of encoding) and code points are both pretty esoteric, but number of symbols is less so. OTOH, SMS has strict data and encoding limits, and people managed with that; also provisioning byte storage for grapheme limited fields is hard: some graphemes use a ton of code points, family emoji and zalgo text are clear examples.

Re: Emoji.length == 2

#67

Post by mikeash »

darkengine wrote: post #52
Manishearth wrote: post #41

Earlier quoted context omitted.

> The thing that frustrates me the most about Unicode emoji is the astounding number of combining characters. For combining characters in written languages, you can do an NFC normalization and, with moderate success, get a 1 codepoint = 1 grapheme mapping, but "Emoji 2.0" introduced some ridiculous emoji compositions with the ZWJ character. No, no, no, NO. Please stop spreading this bit of misinformation. Emoji has n…

That is why I said "with moderate success". It's not 100% reliable, but mostly good enough for basic cases. Twitter, for example, used to do an NFC normalization and count codepoints to enforce the 140 charcater limit, and this was close enough to or exactly right for enforcing 140 graphemes in probably 98% of text on Twitter. They can no longer do this because some new emoji can consume 7 codepoints for a single gra…

So, it was always wrong, but they didn't notice until the "exceptional" cases became common enough in their own use.

This, to me, is an argument for emoji being a good thing for string-handling code. The fact that they're common means that software creators are much more likely to notice when they've written bad string-handling code.

Whatever platform you're using should have an API call for counting grapheme clusters. It may be more complex behind the scenes, but as an ordinary programmer it should be no more difficult to do it correctly than it is to do it wrong.

Re: Emoji.length == 2

#69

Post by PeterisP »

sdegutis wrote: post #4
phkahler wrote: post #2

Unicode is fucked. All these bullshit emojis remind me of the 1980s when ASCII was 7 bits but every computer manufacturer (Atari, Commodore, Apple, IBM, TI, etc...) made their own set of characters for the 128 values of a byte beyond ASCII. Of course Unicode is a global standard so your pile-of-poop emoji will still be a pile-of-poop on every device even if the amount of steam is different for some people. It's beyon…

Language is inherently complex, there's no way to solve this in any "cleaner" way than what we already came up with. Unfortunately the best way forward is to build up what we already have and cover all the warts with wrapper functions/libraries.

Well, there is one way, we can simplify and standardize format of language. Unfortunately that requires generations of "reeducation", so it's not a viable solution in the short term - but it does seem possible that this is where languages are going in the next few centuries, as globalization, easier travel and more interrelated communities are likely to result in slow, gradual convergence to less languages as many of the current 6000+ languages cease to be used in practice.

Re: Emoji.length == 2

#70

Post by yongjik »

duskwuff wrote: post #13
melloclello wrote: post #9

Earlier quoted context omitted.

Man, imagine if you could compose chinese characters out of radicals like this. I'm not sure if that would be a good thing or a bad thing.

The one that still surprises me is Hangul (Korean script). Hangul characters are made of 24 basic characters (jamo) which represent consonant and vowel sounds, which are composed into Hangul characters representing syllables. Unicode has a block for Hangul jamo, but they aren't used in typical text. Instead, Hangul are presented using a massive 11K-codepoint block of every possible precomposed syllable. ¯\_(ツ)_/¯

I believe that was a necessary compromise to use Hangul on any software not authored by Koreans.

"These are characters from a country you've never been to. Each three-byte sequence (assuming UTF-8) corresponds to a square-shaped character." --> Easy for everyone to understand, and less chance of screwup (as long as the software supports any Unicode at all).

"These should be decomposed into sequences of two or three characters, each three bytes long, and then you need a special algorithm to combine them into a square block." --> This pretty much means the software must be developed with Korean users in mind (or someone must heroically go through every part of the code dealing with displaying text), otherwise we might as well assume that it's English-only.

Well, now the equation might be different, as more and more software are developed by global companies and there are more customers using scripts with complicated combining diacritics, but that wasn't the case when Hangul was added to Unicode.

For example: if NFD works properly, the first two characters below should look identical, and the third should show a "defective" character that looks like the first two except without the circle (ㅇ). It doesn't work in gvim (it fails to consider the second/third example as a single character), Chrome in Linux, or Firefox in Linux.

은 은 ᅟᅳᆫ

Of course, if it were the only method of encoding Korean, then the support would have been better, but it would've still required a lot of work by everyone.

Post reply on HN