Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

51–60 of 162 posts

Re: Why we can't process Emoji anymore

#51

Post by [deleted] »

pjscott wrote: post #39
pixelcort wrote: post #18

Earlier quoted context omitted.

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

How many tools have 3-byte limits on UTF-8? The only one I can think of right now is MySQL. (The workaround is to specify the utf8mb4 character set. This is MySQL's cryptic internal name for "actually doing UTF-8 correctly.")

[deleted]

Re: Why we can't process Emoji anymore

#52

Post by kstenerud »

Small nitpick, but Objective-C does not require a particular string encoding internally. In Mac OS and iOS, NSString uses one of the cfinfo flags to specify whether the internal representation is UTF-16 or ASCII (as a space-saving mechanism).

Re: Why we can't process Emoji anymore

#53

Post by evincarofautumn »

Failures in Unicode support seem usually to result from the standard’s persistently shortsighted design—well intentioned and carefully considered though it undoubtedly is. It’s a “good enough” solution to a very difficult problem, but I wonder if we won’t see Unicode supplanted in the next decade.

All that aside: emoji should not be in Unicode. Fullstop.

Re: Why we can't process Emoji anymore

#55

Post by Cogito »

beaumartinez wrote: post #47

This is dated January 2012. By the looks of things, this was fixed in March 2012[1] [1] https://code.google.com/p/v8/issues/detail?id=761#c33

I wonder if this has been rolled into Node yet.

[edit] Node currently uses V8 version 3.11.10.25, which was released after this fix was made, but not sure if the fix was merged to trunk

[edit2] actually, looks like it has, though I can't identify the merge commit

Re: Why we can't process Emoji anymore

#56

Post by othermaciej »

ender7 wrote: post #19

Apropos: http://mathiasbynens.be/notes/javascript-encoding TL;DR: - Javascript engines are free to internally represent strings as either UCS-2 or UTF-16. Engines that choose to go USC-2 tend to replace all glyphs outside of the BMP with the replacement char (U+FFFD). Firefox, IE, Opera, and Safari all do this (with some inconsistencies). - However , from the point of view of the actual JS code that gets executed, st…

Safari uses UTF-16, not UCS-2. I believe this is true of other browsers as well. Otherwise this would render the replacement char, but it doesn't, it renders correctly:

javascript:var x = '𝌆';document.write(x);

Re: Why we can't process Emoji anymore

#57

Post by codeka »

potatolicious wrote: post #21
pixelcort wrote: post #18

Earlier quoted context omitted.

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

Honest question: but isn't that just a broken implementation (and a very obvious brokenness at that)? It seems to me there's a big difference between someone not coding to the standard, and the standard making your taks impossible.

The same could be said of UTF-16 implementations that don't support surrogate pairs.

Re: Why we can't process Emoji anymore

#58

Post by [deleted] »

pixelcort wrote: post #18
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

[deleted]

Re: Why we can't process Emoji anymore

#60

Post by Tsagadai »

pixelcort wrote: post #18
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

File bugs with those tools. These sort of issues should have been sorted years ago and any program that can't do 3+ byte character encodings should be named and shamed.
Post reply on HN