Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

131–140 of 162 posts

Re: Why we can't process Emoji anymore

#131

Post by masklinn »

cjensen wrote: post #41

UCS-16 is only used by programs which jumped the gun and implemented Unicode before it was all done. (It was 16 bits for awhile with Asian languages sharing code points so that the font in use determined whether the text was displayed as Chinese vs Japanese vs. etc). What Century was V8 written in that they thought UCS-16 was an acceptable thing to implement? Good rule of thumb for implementers: get over it and use 3…

> UCS-16 is only used by programs which jumped the gun and implemented Unicode before it was all done.

There's no such thing as "all done", Unicode 1.0 was 16 bit, Unicode 6 was released recently.

Re: Why we can't process Emoji anymore

#132

Post by masklinn »

eps wrote: post #33
pixelcort wrote: post #18

Earlier quoted context omitted.

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

Which tools? Honest question, as the three byte limit seems rather arbitrary and no more logical than, say, a four byte one.

It's not 3 bytes so much as 16 bits, aka Unicode 1.0 limits. Which turn into 3 bytes in UTF-8.

Re: Why we can't process Emoji anymore

#134

Post by masklinn »

derleth wrote: post #32
pixelcort wrote: post #18

Earlier quoted context omitted.

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

And the problem with UTF-16 is that a lot of applications can't handle surrogate pairs, except a lot of Emoji are above the BMP, aren't they? So why is this a bigger deal for UTF-8 than UTF-16?

> except a lot of Emoji are above the BMP, aren't they?

All of the Unicode 6.0 emoji are.

Re: Why we can't process Emoji anymore

#135

Post by masklinn »

Daniel_Newby wrote: post #94
dietrichepp wrote: post #64

Earlier quoted context omitted.

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

> Why do you want to count Unicode characters? Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information. > Why do you care if it is fast to do so? Efficient full text search that can ignore decorative combining characters.

> Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information.

Except these parts of the system have to work on unicode glyphs (rendered characters) which will span multiple codepoints anyway, so counting codepoints remains pointless. The only use it has is knowing how much codepoints you have. Yay.

Re: Why we can't process Emoji anymore

#136

Post by masklinn »

est wrote: post #112
cmccabe wrote: post #110

Earlier quoted context omitted.

http://stackoverflow.com/questions/4579215/cross-platform-it...

Yes? `System.Globalizatiion` or `ICU` can count grapheme, what's your point? Those libraries are equivalent to normalize( utf16 `0x00 0x41 0x03 0x08`) == length 1 Back to my top comment, I stated that UCS2 counts faster than UTf8 internally, because every BMP code point is just two bytes, what's wrong here? If variable-length is so good why py3k is using UCS-4 internally? (Wich means every character is exactly 32 bit…

> Back to my top comment, I stated that UCS2 counts faster than UTf8 internally

The part cmccabe tries to explain, and which you repeatedly fail to understand, is that UCS2 counts unicode code points faster than UTF-8, which is completely useless because "characters" (what the end-user sees as a single sub-unit of text) often spans multiple codepoints, so counting codepoints is essentially a recipe for broken code and nothing else.

> If variable-length is so good why py3k is using UCS-4 internally?

It's not. Before 3.3 it used either UCS2 or UCS4 internally, as did Python 2, since Python 3.3 it switches the internal encoding on the fly.

> Wich means every character is exactly 32 bits. There, I said character again.

yeah and you're wrong again.

Re: Why we can't process Emoji anymore

#137

Post by masklinn »

driverdan wrote: post #4

If you search for V8 UCS-2 you'll find a lot of discussion on this issue dating back at least a few years. There are ways to work around V8's lack of support for surrogate pairs. See this V8 issue for ideas: https://code.google.com/p/v8/issues/detail?id=761 My question is why does V8 (or anything else) still use UCS-2?

> My question is why does V8 (or anything else) still use UCS-2?

Because the ES spec defines a string as a sequence of UTF-16 code units (aka UCS-2-with-visible-surrogates), because as many others (e.g. Java) the language's strings were created during/inherited from Unicode 1.0 which fit in 16 bits (UTF-16 is a retrofitting of Unicode 1.0 fixed-width to accomodate the full range of later unicode version by adding surrogate pairs)

Re: Why we can't process Emoji anymore

#138

Post by masklinn »

dale-cooper wrote: post #130

The UCS-2 heritage is kind of annoying. In java for example, chars (the primitive type, which the Character class just wraps) are 16 bits. So one instance of a Character may not be a full "character" but rather a part of a surrogate pair. This creates a small gotcha where the length of a string might not be the same as the amount of characters it has. And that you just cant split/splice a Character array naively (bec…

Which, at the end of the day, doesn't really matter since a code point is not a "character" in the sense of "the smallest unit of writing" (as interpreted by an end-user): many "characters" may (depending on the normalization form) or will (jamo) span multiple codepoints. Splitting on a character array is always broken, regardless of surrogate pairs.

Re: Why we can't process Emoji anymore

#139

Post by masklinn »

bbotond wrote: post #28
pjscott wrote: post #6

The quick summary, for people who don't like ignoring all those = signs, is that V8 uses UCS-2 internally to represent strings, and therefore can't handle Unicode characters which lie outside the Basic Multilingual Plane -- including Emoji.

Honestly that's a shame.

It was fixed back in March though.
Post reply on HN