Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

111–120 of 162 posts

Re: Why we can't process Emoji anymore

#111

Post by meatmanek »

Daniel_Newby wrote: post #94
dietrichepp wrote: post #64

Earlier quoted context omitted.

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

> Why do you want to count Unicode characters? Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information. > Why do you care if it is fast to do so? Efficient full text search that can ignore decorative combining characters.

> Text editing and rendering

Unless you're working entirely in fixed point characters (and you probably aren't, given that even fixed-width systems like terminal emulators use double-wide glyphs sometimes), you need to know the value of each character to know its width. That involves the same linear scan over the string that is required to calculate the number of glyphs in a variable-width encoding.

Re: Why we can't process Emoji anymore

#112

Post by est »

cmccabe wrote: post #110
est wrote: post #106

Earlier quoted context omitted.

Name one Unicode implementation which shows utf16 `0x00 0x41 0x03 0x08` as length 1. U+4100 U+0803 is two code points by defintion. Thus length == 2.

http://stackoverflow.com/questions/4579215/cross-platform-it...

Yes? `System.Globalizatiion` or `ICU` can count grapheme, what's your point?

Those libraries are equivalent to normalize( utf16 `0x00 0x41 0x03 0x08`) == length 1

Back to my top comment, I stated that UCS2 counts faster than UTf8 internally, because every BMP code point is just two bytes, what's wrong here? If variable-length is so good why py3k is using UCS-4 internally? (Wich means every character is exactly 32 bits. There, I said character again.)

Re: Why we can't process Emoji anymore

#113

Post by pjscott »

Daniel_Newby wrote: post #94
dietrichepp wrote: post #64

Earlier quoted context omitted.

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

> Why do you want to count Unicode characters? Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information. > Why do you care if it is fast to do so? Efficient full text search that can ignore decorative combining characters.

How does fast character counting help with full-text search?

Re: Why we can't process Emoji anymore

#114

Post by pbiggar »

cmccabe wrote: post #105
pbiggar wrote: post #81

A couple of reasons why it makes sense for V8 and other vendors to use UCS2: - The spec says UCS2 or UTF16. Those are the only options. - UCS2 allows random access to characters, UTF-16 does not. - Remember how the JS engines were fighting for speed on arbitrary benchmarks, and nobody cared about anything else for 5 years? UCS2 helps string benchmarks be fast! - Changing from UCS2 to UTF-16 might "break the web", som…

UCS2 allows random access to characters, UTF-16 does not. I'm not sure if that's really true. On IBM's site, they define 3 levels of UCS-2, only one of which excludes "combining characters" (really code points). http://pic.dhe.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%... If you have combining characters, then you can't simply take the number of bytes and divide by 2 to get the number of letters. If you don't have…

OK, I cracked into the V8 source to take a look at what actually happens. It looks like the implementation does use random access for two-byte strings. However, it also uses multiple multiple string implementations (ASCII, 2 byte strings, "consString" (I presume some kind of Rope), "Sliced Strings" (sounds like a rope again, but might be shared storage of the string contents for immutable strings)), so they could likely use other implementations with whatever properties they choose.

See https://github.com/v8/v8/blob/3ff861bbbb62a6c0078e042d8077b2... and https://github.com/v8/v8/blob/3ff861bbbb62a6c0078e042d8077b2....

Re: Why we can't process Emoji anymore

#115

Post by jonny_eh »

speednoise wrote: post #108
fwr wrote: post #25

Earlier quoted context omitted.

On iOS, using any non-basic Latin character in SMS makes it switch to 16 bit, even when there is no reason for that to happen. It's a thing that most foreign language speakers must live with. By doing this full of excuses write-up, this guy wasted a substantial amount of time that he could have spent better researching the issue. Your consumer doesn't care that Emoji is this much or that much bits, it doesn't matter…

This was an internal email: https://medium.com/tech-talk/1aff50f34fc

So Node.js already fixed the issue, nice!

Re: Why we can't process Emoji anymore

#116

Post by notJim »

oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

This comment is somewhat misleading. The issue at hand is orthogonal to any of the benefits of UTF-8 over UTF-16 (which are real, UTF-8 is great, you should use it.)

4-byte characters in UTF-8 are just as rare as surrogate pairs are just as rare in UTF-16, because they both are used to represent non-BMP characters. As a result, there is software that handles 3-byte characters (i.e., a huge percentage of what you'll ever see), but doesn't handle 4-byte characters.

MySQL is a high-profile example of software which, until recently, had this problem: http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8m....

Re: Why we can't process Emoji anymore

#117

Post by notJim »

est wrote: post #62
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

I don't really think you can argue that UCS2 is better than UTF-8, because UCS2 is simply broken. It's not a reliable way to encode unicode characters. It's like arguing that my Ferrari which is currently on fire is a better car than your Lamborghini (which is not.) I mean, they do each have their merits, but a flaming car is not useful to anyone.

I think this is probably semantics, but I just wanted to point that out in case anyone is confused, which would be understandable because this shit is whack.

Re: Why we can't process Emoji anymore

#118

Post by notJim »

ender7 wrote: post #19

Apropos: http://mathiasbynens.be/notes/javascript-encoding TL;DR: - Javascript engines are free to internally represent strings as either UCS-2 or UTF-16. Engines that choose to go USC-2 tend to replace all glyphs outside of the BMP with the replacement char (U+FFFD). Firefox, IE, Opera, and Safari all do this (with some inconsistencies). - However , from the point of view of the actual JS code that gets executed, st…

I'm relatively comfortable with this stuff, but I am confused by your response.

First you say that engines will "internally" replace non-BMP glyphs with the replacement character, but then you give an example that seems to work fine (and I think would work fine as long as you don't cut that character in half, or try to inspect its character code without doing the proper incantations[1].)

So, I guess what I'm asking is, at what point does the string become "internal", such that the engine will replace the character with the replacement character?

[1]: As given in the article you linked to.

Re: Why we can't process Emoji anymore

#119

Post by xentronium »

csense wrote: post #96
est wrote: post #62

Earlier quoted context omitted.

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

Issues like this are why I hate internationalization. If it was simple as making everything Unicode and it Just Working, it would be possible. But the number of difficulties and problems I've seen have made me decide -- and tell everyone I know -- to avoid dealing with internationalization if you value your sanity. Issues discussed here: * Different incompatible variable-length encodings * Broken implementations * Ch…

> Should my C programs handle the possibility that sizeof(char) != 1? Or at least check for this case and spit out a warning or error?

sizeof(char) == 1 by definition.

Re: Why we can't process Emoji anymore

#120

Post by andolanra »

Daniel_Newby wrote: post #94
dietrichepp wrote: post #64

Earlier quoted context omitted.

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

> Why do you want to count Unicode characters? Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information. > Why do you care if it is fast to do so? Efficient full text search that can ignore decorative combining characters.

If you implement naïve Aho-Corasick text search over one-byte characters, it works without modification on UTF-8 text. It does not ignore combining characters, but UCS-2 also features combining characters (c.f. other comments in this same thread), so no matter what encoding you use, you must first normalize the Unicode text and the search string before you compare for equivalence (or compatibility, which is a looser notion than equality for Unicode code point sequences.)
Post reply on HN