Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

151–160 of 162 posts

Re: Why we can't process Emoji anymore

#151

Post by snogglethorpe »

notJim wrote: post #116
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

This comment is somewhat misleading. The issue at hand is orthogonal to any of the benefits of UTF-8 over UTF-16 (which are real, UTF-8 is great, you should use it.) 4-byte characters in UTF-8 are just as rare as surrogate pairs are just as rare in UTF-16, because they both are used to represent non-BMP characters. As a result, there is software that handles 3-byte characters (i.e., a huge percentage of what you'll e…

The problem is that handling 1 unit is very different from 2+ units, in terms of coding patterns, whereas 3 is not so different from 4+. In the latter case there's already probably a loop to handle multiple unit characters, which will in many cases work without change for longer sequences (and if not, probably the code probably requires very little change to do so).

So whereas it's rather common for programs to mis-handle multiple-unit UTF-16 characters, it seems much less likely that programs will mis-handle 4+ unit UTF-8 characters.

Re: Why we can't process Emoji anymore

#152

Post by masklinn »

dale-cooper wrote: post #147
masklinn wrote: post #143

Earlier quoted context omitted.

> It's a leaky abstraction, you shouldn't need to handle something that is tied to the internal representation of strings in the jvm. And I'm saying it doesn't really matter, because unicode codepoints are already a form of "leaky abstraction" which you'll have to handle (in that a read/written "character" does not correspond 1:1 to a codepoint anyway). Unicode is a tentative standardization of historical human produ…

That's quite interesting, i had no idea! What i was hoping for was some kind of term for one character or symbol and use that as a unit, but perhaps it's impossible to create an abstraction like that. I'm curious if a Sanskrit speaker would see each of the codepoints as a symbol or not. Edit: thinking about it, i guess if you asked a Sanskrit speaker how long a word/sentence was, you'd get the answer..

> What i was hoping for was some kind of term for one character or symbol and use that as a unit

There is one, kind-of: "grapheme cluster"[0]. This is the "unit" used by UAX29 to define text segmentation, and aliases to "user-perceived character"[1].

Most languages/API don't really consider them (although they crop up often in e.g. browser bug trackers), let alone provide first-class access to them. One of the very few APIs which actually acknowledges them is Cocoa's NSString — and Apple provides a document explaining grapheme clusters and how they relate to NNString[2] — which has very good unicode support (probably the best I know of, though Factor may have an even better one[3]), and it handles grapheme clusters through providing messages which work on codepoint ranges in an NSString, it doesn't treat clusters as first-class objects.

> i guess if you asked a Sanskrit speaker how long a word/sentence was, you'd get the answer..

Indeed.

[0] http://www.unicode.org/glossary/#grapheme_cluster

[1] http://www.unicode.org/reports/tr29/#Grapheme_Cluster_Bounda...

[2] https://developer.apple.com/library/mac/#documentation/Cocoa...

[3] the original implementor detailed his whole route through creating factor's unicode library, and I learned a lot from it: http://useless-factor.blogspot.be/search/label/unicode

Re: Why we can't process Emoji anymore

#154

Post by clebio »

ladon86 wrote: post #91
clebio wrote: post #44

Somewhat meta, but this would be one where showing subdomain on HN submissions would be nice. The title is vague enough that I assumed it was something to do with _Github_ not processing Emoji (which would be sort of a strange state of affairs...).

Not that strange, Github implements much of the Emoji set using different shortcuts, see the reference here: http://www.emoji-cheat-sheet.com/ Before I read the article I guessed that maybe the icon set had some licensing issues for Github. Luckily, not so! (:smiley:)

That was basically my point. It would be strange if they _stopped_ processing it.

Re: Why we can't process Emoji anymore

#155

Post by Daniel_Newby »

pjscott wrote: post #113
Daniel_Newby wrote: post #94

Earlier quoted context omitted.

> Why do you want to count Unicode characters? Text editing and rendering. Some parts of the system cannot simply treat Unicode text as an opaque binary hunk of information. > Why do you care if it is fast to do so? Efficient full text search that can ignore decorative combining characters.

How does fast character counting help with full-text search?

The best search algorithms can skip ahead upon a mismatch. A variable-length encoding requires branch instructions in the inner loop, leading to pipeline flushes and potentially dramatic slow down.

Re: Why we can't process Emoji anymore

#156

Post by jruderman »

dgreensp wrote: post #74

The specific problems the author describes don't seem to be present today; perhaps they were fixed. That's not to say this conversions aren't a source of issues, just that I don't see any show-stopper problems currently in Node, V8, or JavaScript. In JavaScript, a string is a series of UTF-16 code units, so the smiley face is written '\ud83d\ude04'. This string has length 2, not 1, and behaves like a length-2 string…

The invisible smiley was a font system problem, fixed in Firefox 19 Aurora (assuming you're on Mac).

https://bugzilla.mozilla.org/show_bug.cgi?id=715798

Re: Why we can't process Emoji anymore

#157

Post by threedaymonk »

est wrote: post #62
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

"Character". You keep saying that word …

    >>> len(u'épicé')
    5
    >>> len(u'épicé')
    7

Re: Why we can't process Emoji anymore

#158

Post by est »

masklinn wrote: post #136
est wrote: post #112

Earlier quoted context omitted.

Yes? `System.Globalizatiion` or `ICU` can count grapheme, what's your point? Those libraries are equivalent to normalize( utf16 `0x00 0x41 0x03 0x08`) == length 1 Back to my top comment, I stated that UCS2 counts faster than UTf8 internally, because every BMP code point is just two bytes, what's wrong here? If variable-length is so good why py3k is using UCS-4 internally? (Wich means every character is exactly 32 bit…

> Back to my top comment, I stated that UCS2 counts faster than UTf8 internally The part cmccabe tries to explain, and which you repeatedly fail to understand, is that UCS2 counts unicode code points faster than UTF-8, which is completely useless because "characters" (what the end-user sees as a single sub-unit of text) often spans multiple codepoints, so counting codepoints is essentially a recipe for broken code an…

> yeah and you're wrong again.

http://en.wikipedia.org/wiki/UTF-32

UTF-32 (or UCS-4) is a protocol to encode Unicode characters that uses exactly 32 bits per Unicode code point.

http://www.unicode.org/faq/utf_bom.html#utf32-1

Re: Why we can't process Emoji anymore

#159

Post by dale-cooper »

masklinn wrote: post #152
dale-cooper wrote: post #147

Earlier quoted context omitted.

That's quite interesting, i had no idea! What i was hoping for was some kind of term for one character or symbol and use that as a unit, but perhaps it's impossible to create an abstraction like that. I'm curious if a Sanskrit speaker would see each of the codepoints as a symbol or not. Edit: thinking about it, i guess if you asked a Sanskrit speaker how long a word/sentence was, you'd get the answer..

> What i was hoping for was some kind of term for one character or symbol and use that as a unit There is one, kind-of: "grapheme cluster"[0]. This is the "unit" used by UAX29 to define text segmentation, and aliases to "user-perceived character"[1]. Most languages/API don't really consider them (although they crop up often in e.g. browser bug trackers), let alone provide first-class access to them. One of the very f…

Very interesting, going to read through that guys blog. Thanks for the links!

Re: Why we can't process Emoji anymore

#160

Post by cmccabe »

est wrote: post #127
cmccabe wrote: post #121

Earlier quoted context omitted.

I am suggesting that people read about unicode before designing supposedly cross-platform applications or programming languages. It's not that hard, just different than ASCII.

Since you understand Unicode so well, can you explain dietrichepp's theory that Unicode don't need counting or offsets? http://news.ycombinator.com/item?id=4834931 And why UCS4 (Not variable-length) is chosen in many Unicode implementations? Why wchar_t is always 32bit in posix?

Since you understand Unicode so well, can you explain dietrichepp's theory that Unicode don't need counting or offsets?

Unicode doesn't have "characters." If you talk about characters, all you've succeeded in doing is confusing yourself. Leave characters back in ASCII-land where they belong.

Counting code points is stupid. If you like counting code points, go sit in the corner. You don't understand unicode.

You can count graphemes, but it's not going to be easy. And most of the time, I don't see why you would need to do that.

And why UCS4 (Not variable-length) is chosen in many Unicode implementations? Why wchar_t is always 32bit in posix?

wchat_t is a horrible abomination that begs for death. Nobody should use it. Use UTF-8 instead. I think Python used to use UCS4, but they don't any more. It's a horrible representation because all your strings bloat up by 4x.

Post reply on HN