Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

141–150 of 162 posts

Re: Why we can't process Emoji anymore

#141

Post by dale-cooper »

masklinn wrote: post #138
dale-cooper wrote: post #130

The UCS-2 heritage is kind of annoying. In java for example, chars (the primitive type, which the Character class just wraps) are 16 bits. So one instance of a Character may not be a full "character" but rather a part of a surrogate pair. This creates a small gotcha where the length of a string might not be the same as the amount of characters it has. And that you just cant split/splice a Character array naively (bec…

Which, at the end of the day, doesn't really matter since a code point is not a "character" in the sense of "the smallest unit of writing" (as interpreted by an end-user): many "characters" may (depending on the normalization form) or will (jamo) span multiple codepoints. Splitting on a character array is always broken, regardless of surrogate pairs.

Yes. What i'm saying is that it would feel less error prone if the character object was actually a codepoint. It's a leaky abstraction, you shouldn't need to handle something that is tied to the internal representation of strings in the jvm. Can one "character" span multiple codepoints? Do you have an example of this?

Re: Why we can't process Emoji anymore

#142

Post by jrabone »

prodigal_erik wrote: post #78
jrabone wrote: post #43

Earlier quoted context omitted.

No. Combining characters and NF(K)C/D normalisation rules are a different problem entirely - consider the "heavy metal umlaut" (ie. Spın̈al Tap) where there is no lossless conversion possible - only “n" followed by U+0308

They're facets of the same problem. I shouldn't routinely be dealing with either surrogates or combining marks; unless I have a specific reason, it's only an opportunity to make a mistake that hardly anyone knows how to troubleshoot. "n̈" should be an indivisible string of length one until I need to ask how it would actually be encoded in UTF-16 or whatever.

But that's the point - there is no such character. Given the Unicode consortium have added codepoints for every other bloody thing under the sun, I'm amazed that there isn't one for n-diaresis but there you are.

Add a small number of people who for artistic reasons decide that they want to make life hard (Rinôçérôse I'm looking at you) and you just have to accept that the length of your string might not equal the number of codepoints contained therein...

Re: Why we can't process Emoji anymore

#143

Post by masklinn »

dale-cooper wrote: post #141
masklinn wrote: post #138

Earlier quoted context omitted.

Which, at the end of the day, doesn't really matter since a code point is not a "character" in the sense of "the smallest unit of writing" (as interpreted by an end-user): many "characters" may (depending on the normalization form) or will (jamo) span multiple codepoints. Splitting on a character array is always broken, regardless of surrogate pairs.

Yes. What i'm saying is that it would feel less error prone if the character object was actually a codepoint. It's a leaky abstraction, you shouldn't need to handle something that is tied to the internal representation of strings in the jvm. Can one "character" span multiple codepoints? Do you have an example of this?

> It's a leaky abstraction, you shouldn't need to handle something that is tied to the internal representation of strings in the jvm.

And I'm saying it doesn't really matter, because unicode codepoints are already a form of "leaky abstraction" which you'll have to handle (in that a read/written "character" does not correspond 1:1 to a codepoint anyway). Unicode is a tentative standardization of historical human production, and if you expect that to end up clean and simple you're going to have a hard time.

> Can one "character" span multiple codepoints?

Yes.

> Do you have an example of this?

Devanagari (the script used for e.g. Sanskrit) is full of them. For instance, "sanskrit" is written "संस्कृतम्" [sə̃skɹ̩t̪əm]. If you try to select "characters" in your browser you might get 4 (सं, स्कृ, त and म्) or 5 (सं, स्, कृ, त and म्) or maybe yet another different count, but this is a sequence of 9 codepoints (regardless of the normalization, it's the same in all of NFC, NFD, NFKC and NFKD as far as I can tell):

    स: DEVANAGARI LETTER SA
    ं: DEVANAGARI SIGN ANUSVARA
    स: DEVANAGARI LETTER SA
    ्: DEVANAGARI SIGN VIRAMA
    क: DEVANAGARI LETTER KA
    ृ: DEVANAGARI VOWEL SIGN VOCALIC R
    त: DEVANAGARI LETTER TA
    म: DEVANAGARI LETTER MA
    ्: DEVANAGARI SIGN VIRAMA
Note: I'm not a Sanskrit speaker and I don't actually know devanagari (beyond knowing that it's troublesome for computers, as are jamo) so I can't even tell you how many "symbols" a native reader would see there.

Re: Why we can't process Emoji anymore

#144

Post by speleding »

est wrote: post #65
driverdan wrote: post #4

If you search for V8 UCS-2 you'll find a lot of discussion on this issue dating back at least a few years. There are ways to work around V8's lack of support for surrogate pairs. See this V8 issue for ideas: https://code.google.com/p/v8/issues/detail?id=761 My question is why does V8 (or anything else) still use UCS-2?

because counting 2 bytes is much faster for computers than counting vary 1, 2, 3 or even 4 bytes.

This is not a real issue because counting code points in an UTF8 string is easy too: the encoding is cleverly defined such that you just need to check the number of bytes that have the top bit cleared. Since UTF8 strings are generally shorter it can even be faster than counting UTF-16 if you don't know the length in advance.

Re: Why we can't process Emoji anymore

#145

Post by masklinn »

masklinn wrote: post #133

Wow, the first half of the text is basically full of crap and claims which don't even remotely match reality, and now I'm reaching the technical section which can only get even more wrong.

To whoever the downvoter was: no, seriously. For instance in the first few paragraphs:

* emoji were invented by NTT DoCoMo, not Softbank

* even if that had been right Softbank's copyrighting of their emoji representations has no bearing on NTT and KDDI/au using completely different implementations (and I do mean completely, KDDI/au essentially use tags)

* lack of cooperation is endemic to japanese markets (especially telecoms) and has nothing to do with "ganging up"

* if NTT and au/KDDI wanted to gang up on Softbank you'd think they'd share the same emoji

* you didn't have to run "adware apps" to unlock the emoji keyboard (there were numerous ways to do so from dedicated — and usually quickly nuked – apps to apps "easter eggs" to jailbreak to phone backup edit/restore)

That's barely the first third.

Re: Why we can't process Emoji anymore

#146

Post by speleding »

est wrote: post #62
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

Counting code points in UTF8 can actually be faster than UCS2 in case you do not know the length in advance such as null-terminated strings. The UTF8 encoding is cleverly defined such that to find the number of code points you just need to check the number of bytes that have the top bit cleared. Since UTF8 strings are generally shorter than UCS2 it can be faster in some cases. Either way this is not a serious enough concern to use one encoding over the other.

Re: Why we can't process Emoji anymore

#147

Post by dale-cooper »

masklinn wrote: post #143
dale-cooper wrote: post #141

Earlier quoted context omitted.

Yes. What i'm saying is that it would feel less error prone if the character object was actually a codepoint. It's a leaky abstraction, you shouldn't need to handle something that is tied to the internal representation of strings in the jvm. Can one "character" span multiple codepoints? Do you have an example of this?

> It's a leaky abstraction, you shouldn't need to handle something that is tied to the internal representation of strings in the jvm. And I'm saying it doesn't really matter, because unicode codepoints are already a form of "leaky abstraction" which you'll have to handle (in that a read/written "character" does not correspond 1:1 to a codepoint anyway). Unicode is a tentative standardization of historical human produ…

That's quite interesting, i had no idea! What i was hoping for was some kind of term for one character or symbol and use that as a unit, but perhaps it's impossible to create an abstraction like that.

I'm curious if a Sanskrit speaker would see each of the codepoints as a symbol or not.

Edit: thinking about it, i guess if you asked a Sanskrit speaker how long a word/sentence was, you'd get the answer..

Re: Why we can't process Emoji anymore

#148

Post by mikeash »

pjscott wrote: post #75
mikeash wrote: post #71

Earlier quoted context omitted.

I actually switched my (low traffic, low performance needs) blog comments database from MySQL to SQLite purely because I could not make MySQL and Unicode get along. All I needed was for it to accept and then regurgitate UTF-8 and it couldn't even handle that. I'm sure it can be done, but none of the incantations I tried made it work, and it was ultimately easier for me to switch databases.

As an ugly last resort, you could store Unicode as UTF-8 in BLOB fields. MySQL is pretty good about storing binary data. (I dread the day that I'll have to do something more advanced with Unicode in MySQL than just storing it.)

I no longer recall whether I tried that and failed, or didn't get that far. Seems like a semi-reasonable approach if you don't need the database to be able to understand the contents of that column. But on the other hand, SQLite is working great for my needs too.

Re: Why we can't process Emoji anymore

#149

Post by Kaworu »

notJim wrote: post #118
ender7 wrote: post #19

Apropos: http://mathiasbynens.be/notes/javascript-encoding TL;DR: - Javascript engines are free to internally represent strings as either UCS-2 or UTF-16. Engines that choose to go USC-2 tend to replace all glyphs outside of the BMP with the replacement char (U+FFFD). Firefox, IE, Opera, and Safari all do this (with some inconsistencies). - However , from the point of view of the actual JS code that gets executed, st…

I'm relatively comfortable with this stuff, but I am confused by your response. First you say that engines will "internally" replace non-BMP glyphs with the replacement character, but then you give an example that seems to work fine (and I think would work fine as long as you don't cut that character in half, or try to inspect its character code without doing the proper incantations[1].) So, I guess what I'm asking i…

I dare not try and reexplain the discussion in this bug report as my understanding feels insufficient, but the entire discussion at http://code.google.com/p/v8/issues/detail?id=761#c14 (note, I've linked to the 14th commment in the discussion, but there's more interesting stuff above) talks about it. At the core is a distinction between v8's internal representation of strings and it's API vs. what a browser engine which embeds v8 might do.

Re: Why we can't process Emoji anymore

#150

Post by mikeash »

jrabone wrote: post #49
eps wrote: post #33

Earlier quoted context omitted.

Which tools? Honest question, as the three byte limit seems rather arbitrary and no more logical than, say, a four byte one.

It is totally arbitrary - there's no reason you can't have degenerate 6-byte encodings, and compliant decoders should cope with them. See Marcus Kuhn's excellent UTF-8 decoder torture test page linked elsewhere in this thread.

Actually, a truly compliant UTF-8 decoder should reject all degenerate forms, because they can be used to bypass validation checks, and the spec does not allow such forms. A four-byte UTF-8 sequence is sufficient to represent any Unicode code point (and then some), so no compliant decoder should accept any sequences of 5 or 6 bytes.

Note that Kuhn's torture test page deliberately includes a lot of invalid sequences in order to make sure that they're gracefully handled. Section 4 is dedicated to degenerate encodings like this.

Post reply on HN