Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

71–80 of 162 posts

Re: Why we can't process Emoji anymore

#71

Post by mikeash »

jrabone wrote: post #45
pjscott wrote: post #39

Earlier quoted context omitted.

How many tools have 3-byte limits on UTF-8? The only one I can think of right now is MySQL. (The workaround is to specify the utf8mb4 character set. This is MySQL's cryptic internal name for "actually doing UTF-8 correctly.")

MySQL is one of the worst offenders for broken Unicode and collation problems arising therein. Neither it nor JavaScript deserve consideration for problems that need robust Unicode handling.

I actually switched my (low traffic, low performance needs) blog comments database from MySQL to SQLite purely because I could not make MySQL and Unicode get along. All I needed was for it to accept and then regurgitate UTF-8 and it couldn't even handle that. I'm sure it can be done, but none of the incantations I tried made it work, and it was ultimately easier for me to switch databases.

Re: Why we can't process Emoji anymore

#72

Post by pjscott »

dietrichepp wrote: post #64
est wrote: post #62

Earlier quoted context omitted.

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

Be careful not to confuse UTF-16 and UCS-2. UTF-16 is a variable-width encoding, so using it doesn't actually make counting anything easier. UCS-2 is fixed-width, and evil. With UCS-2 you can easily count the number of code points, as long as they fall in the BMP (!), but this is not the same as counting graphemes.

Re: Why we can't process Emoji anymore

#73

Post by mikeash »

[deleted] wrote: post #67
dietrichepp wrote: post #64

Earlier quoted context omitted.

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

[deleted]

Such an API could easily indicate bytes in a UTF-8 string instead.

Re: Why we can't process Emoji anymore

#74

Post by dgreensp »

The specific problems the author describes don't seem to be present today; perhaps they were fixed. That's not to say this conversions aren't a source of issues, just that I don't see any show-stopper problems currently in Node, V8, or JavaScript.

In JavaScript, a string is a series of UTF-16 code units, so the smiley face is written '\ud83d\ude04'. This string has length 2, not 1, and behaves like a length-2 string as far as regexes, etc., which is too bad. But even though you don't get the character-counting APIs you might want, the JavaScript engine knows this is a surrogate pair and represents a single code point (character). (It just doesn't do much with this knowledge.)

You can assign '\ud83d\ude04' to document.body.innerHTML in modern Chrome, Firefox, or Safari. In Safari you get a nice Emoji; in stock Chrome and Firefox, you don't, but the empty space is selectable and even copy-and-pastable as a smiley! So the character is actually there, it just doesn't render as a smiley.

The bug that may have been present in V8 or Node is: what happens if you take this length-2 string and write it to a UTF8 buffer, does it get translated correctly? Today, it does.

What if you put the smiley directly into a string literal in JS source code, not \u-escaped? Does that work? Yes, in Chrome, Firefox, and Safari.

Re: Why we can't process Emoji anymore

#75

Post by pjscott »

mikeash wrote: post #71
jrabone wrote: post #45

Earlier quoted context omitted.

MySQL is one of the worst offenders for broken Unicode and collation problems arising therein. Neither it nor JavaScript deserve consideration for problems that need robust Unicode handling.

I actually switched my (low traffic, low performance needs) blog comments database from MySQL to SQLite purely because I could not make MySQL and Unicode get along. All I needed was for it to accept and then regurgitate UTF-8 and it couldn't even handle that. I'm sure it can be done, but none of the incantations I tried made it work, and it was ultimately easier for me to switch databases.

As an ugly last resort, you could store Unicode as UTF-8 in BLOB fields. MySQL is pretty good about storing binary data. (I dread the day that I'll have to do something more advanced with Unicode in MySQL than just storing it.)

Re: Why we can't process Emoji anymore

#76

Post by marshray »

[deleted] wrote: post #67
dietrichepp wrote: post #64

Earlier quoted context omitted.

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

[deleted]

You will never define an unambiguous way to index into a Unicode string by "characters" with less than 10 pages of specification. When two sides of the API interpret this contract differently, a security vulnerability is a very possible outcome.

Use unambiguous delimiters, JSON, XML, or even byte offsets into normalized UTF-8 instead. But don't do this please.

Re: Why we can't process Emoji anymore

#77

Post by marshray »

throwaway54-762 wrote: post #12
akie wrote: post #5

Earlier quoted context omitted.

TLDR: The V8 engine can't (supposedly) encode Unicode codepoints that are over 16-bits in length, because it uses the UCS-2 encoding.

TLDR: v8 "sucks" (and doesn't support Unicode code points outside of the lowest ~64k characters). Edit: v8 in general is pretty cool, but not supporting Unicode outside UCS-2 is pretty bad.

Most apps seem to "support" surrogate pairs by simply not being aware of them at all.

Good on the V8 developers for recognizing these conditions that their code didn't fully handle and refusing to muddle on through with broken processing.

Re: Why we can't process Emoji anymore

#78

Post by prodigal_erik »

jrabone wrote: post #43
prodigal_erik wrote: post #13

Earlier quoted context omitted.

Unicode 2.0 added surrogate pairs in 1996. Unfortunately, the first versions of both Java and JavaScript predated this and got strings horribly wrong, and now any conforming implementation of either is required to suck. The Right Thing would be for almost everyone to work with only combining character sequences, except for a rare few who need to know how to dissect one into its codepoints and reassemble them correctl…

No. Combining characters and NF(K)C/D normalisation rules are a different problem entirely - consider the "heavy metal umlaut" (ie. Spın̈al Tap) where there is no lossless conversion possible - only “n" followed by U+0308

They're facets of the same problem. I shouldn't routinely be dealing with either surrogates or combining marks; unless I have a specific reason, it's only an opportunity to make a mistake that hardly anyone knows how to troubleshoot. "n̈" should be an indivisible string of length one until I need to ask how it would actually be encoded in UTF-16 or whatever.

Re: Why we can't process Emoji anymore

#79

Post by thristian »

jrabone wrote: post #29
thristian wrote: post #22

Earlier quoted context omitted.

I don't think it's specifically a 3-byte limit, I think it's just that lots of tools decode UTF-8 into UCS-2 internally instead of UTF-16.

That is quite clearly broken, and any tool that does so should be fixed or dumped. This is not new, and Marcus Kuhn had made UTF8 test resources available for years at http://www.cl.cam.ac.uk/~mgk25/unicode.html

I believe that MySQL is one such tool. In recent versions you can work around it by asking for the encoding "utf8mb4" instead of "utf8", but I think they have to be quite recent.

So yes, another way in which MySQL is quite clearly broken.

Re: Why we can't process Emoji anymore

#80

Post by est »

dietrichepp wrote: post #64
est wrote: post #62

Earlier quoted context omitted.

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

> Why do you want to count Unicode characters?

Because, there are other countries which use more than English language?

I fucking hate you ascii-centric ignorant morons sometimes, you know, for example

- display welcome message character by character fro left to right

- Extract the first character because it's always the surname

- catch two non-ascii keyword and find its index in a string

In the first example, should I just put byte by byte, which displays as garbage, and suddenly three bytes become a recognizable character?

Post reply on HN