Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

21–30 of 162 posts

Re: Why we can't process Emoji anymore

#21

Post by potatolicious »

pixelcort wrote: post #18
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

Honest question: but isn't that just a broken implementation (and a very obvious brokenness at that)? It seems to me there's a big difference between someone not coding to the standard, and the standard making your taks impossible.

Re: Why we can't process Emoji anymore

#22

Post by thristian »

pixelcort wrote: post #18
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

I don't think it's specifically a 3-byte limit, I think it's just that lots of tools decode UTF-8 into UCS-2 internally instead of UTF-16.

Re: Why we can't process Emoji anymore

#24

Post by dmethvin »

marshray wrote: post #15
freedrull wrote: post #11

Why on earth would the people who wrote V8 use UCS-2? What about alternative JS runtimes?

Because Unicode was sold to the world's software developers as a fixed-width encoding claiming 16 bits would be all we'd ever need.

Yes, and several C/C++ conventions and types seemed to make that a safe choice, for example wchar_t. Let's face it, collectively we really screwed this one up. It's the biggest mistake since Microsoft chose the backslash as a path separator in DOS 2.0.

Re: Why we can't process Emoji anymore

#25

Post by fwr »

praptak wrote: post #9

Sometimes you need to know about encodings, even if you're just a consumer. Putting just one non 7-bit character in your SMS message will silently change its encoding from 7-bit (160 chars) to 8-bit (140 chars) or even 16 bit (70 chars) which might make the phone split it into many chunks. The resulting chunks are billed as separate messages.

On iOS, using any non-basic Latin character in SMS makes it switch to 16 bit, even when there is no reason for that to happen. It's a thing that most foreign language speakers must live with.

By doing this full of excuses write-up, this guy wasted a substantial amount of time that he could have spent better researching the issue. Your consumer doesn't care that Emoji is this much or that much bits, it doesn't matter for him that you're running your infrastructure on poorly chosen software - there is absolutely no excuse for not supporting this in a native iOS app, especially now that Emoji is so widely used and deeply integrated in iOS.

How is that a problem they are focusing on, anyway, when their landing page features awful, out of date mockups of the app? (not even actual screenshots - notice the positions of menu bar items) They are also featuring Emoji in every screenshot - ending support might be a fresh development, but I still find that ironic.

Re: Why we can't process Emoji anymore

#27

Post by hkmurakami »

>Wow, you read though all of that? You rock. I'm humbled that you gave me so much of your attention.

That was actually really fun to read, even as a now non-technical guy. I can't put a finger on it, but there was something about his style that gave off a really friendly vibe even through all the technical jargon. That's a definite skill!

Re: Why we can't process Emoji anymore

#28

Post by bbotond »

pjscott wrote: post #6

The quick summary, for people who don't like ignoring all those = signs, is that V8 uses UCS-2 internally to represent strings, and therefore can't handle Unicode characters which lie outside the Basic Multilingual Plane -- including Emoji.

Honestly that's a shame.

Re: Why we can't process Emoji anymore

#29

Post by jrabone »

thristian wrote: post #22
pixelcort wrote: post #18

Earlier quoted context omitted.

The problem with UTF-8 is that lots of tools have 3 byte limits, and characters like Emoji take up 4 bytes in UTF-8.

I don't think it's specifically a 3-byte limit, I think it's just that lots of tools decode UTF-8 into UCS-2 internally instead of UTF-16.

That is quite clearly broken, and any tool that does so should be fixed or dumped. This is not new, and Marcus Kuhn had made UTF8 test resources available for years at http://www.cl.cam.ac.uk/~mgk25/unicode.html

Re: Why we can't process Emoji anymore

#30

Post by magic_haze »

dmethvin wrote: post #24
marshray wrote: post #15

Earlier quoted context omitted.

Because Unicode was sold to the world's software developers as a fixed-width encoding claiming 16 bits would be all we'd ever need.

Yes, and several C/C++ conventions and types seemed to make that a safe choice, for example wchar_t. Let's face it, collectively we really screwed this one up. It's the biggest mistake since Microsoft chose the backslash as a path separator in DOS 2.0.

It was actually IBM's fault: they used '/' to denote CLI args in the apps they wrote for DOS 1.0, which didn't have any concept of directories. From Larry Osterman's blog [1]:

> Here's a little known secret about MS-DOS. The DOS developers weren't particularly happy about this state of affairs - heck, they all used Xenix machines for email and stuff, so they were familiar with the *nix command semantics. So they coded the OS to accept either "/" or "\" character as the path character (this continues today, btw - try typing "notepad c:/boot.ini" on an XP machine (if you're an admin)). And they went one step further. They added an undocumented system call to change the switch character. And updated the utilities to respect this flag.

[1] http://blogs.msdn.com/b/larryosterman/archive/2005/06/24/432...

Post reply on HN