Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

11–20 of 162 posts

Re: Why we can't process Emoji anymore

#12

Post by throwaway54-762 »

akie wrote: post #5
sneak wrote: post #3

TLDR: node sucks

TLDR: The V8 engine can't (supposedly) encode Unicode codepoints that are over 16-bits in length, because it uses the UCS-2 encoding.

TLDR: v8 "sucks" (and doesn't support Unicode code points outside of the lowest ~64k characters).

Edit: v8 in general is pretty cool, but not supporting Unicode outside UCS-2 is pretty bad.

Re: Why we can't process Emoji anymore

#13

Post by prodigal_erik »

tptacek wrote: post #8
sneak wrote: post #3

TLDR: node sucks

It's v8's fault, and v8 does not suck.

Unicode 2.0 added surrogate pairs in 1996. Unfortunately, the first versions of both Java and JavaScript predated this and got strings horribly wrong, and now any conforming implementation of either is required to suck. The Right Thing would be for almost everyone to work with only combining character sequences, except for a rare few who need to know how to dissect one into its codepoints and reassemble them correctly (just as people don't normally need to extract high or low bits from an ASCII character).

Re: Why we can't process Emoji anymore

#14

Post by languagehacker »

We seem to be seeing this more and more with Node-based applications. It's a symptom of the platform being too immature. This is why you shouldn't adopt these sorts of stacks unless there's some feature they provide that none of the more mature stacks support yet. And even then, you should probably ask yourself if you really need that feature.

Re: Why we can't process Emoji anymore

#16

Post by oofabz »

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed.

UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCII compatibility, size efficient).

Re: Why we can't process Emoji anymore

#18

Post by pixelcort »

oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

Re: Why we can't process Emoji anymore

#19

Post by ender7 »

Apropos: http://mathiasbynens.be/notes/javascript-encoding

TL;DR:

- Javascript engines are free to internally represent strings as either UCS-2 or UTF-16. Engines that choose to go USC-2 tend to replace all glyphs outside of the BMP with the replacement char (U+FFFD). Firefox, IE, Opera, and Safari all do this (with some inconsistencies).

- However, from the point of view of the actual JS code that gets executed, strings are always UCS-2 (sort of). In UTF-16, code points outside the BMP are encoded as surrogate pairs (4 bytes). But -- if you have a Javascript string that contains such a character, it will be treated as two consecutive 2-byte characters.

  var x = '𝌆';
  x.length; // 2
  x[0];     // \uD834
  x[1];     // \uDF06
Note that if you insert said string into the DOM, it will still render correctly (you'll see a single character instead of two ?s).

Re: Why we can't process Emoji anymore

#20

Post by thristian »

marshray wrote: post #15
freedrull wrote: post #11

Why on earth would the people who wrote V8 use UCS-2? What about alternative JS runtimes?

Because Unicode was sold to the world's software developers as a fixed-width encoding claiming 16 bits would be all we'd ever need.

Yes it was, in 1991 when NT and Java and Cocoa were new or under development. In 1996, Unicode 2.0 came out with surrogate pairs and astral planes, and Unicode was no longer a 16 bit encoding.

I'm pretty sure work on V8 started after 1996.

More likely is the idea that the authors of V8 felt that UCS-2 was an acceptable speed/correctness trade-off.

Post reply on HN