Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we can't process Emoji anymore

gist.github.com

Post reply on HN
Print view View original

61–70 of 162 posts

Re: Why we can't process Emoji anymore

#61

Post by dmethvin »

magic_haze wrote: post #30
dmethvin wrote: post #24

Earlier quoted context omitted.

Yes, and several C/C++ conventions and types seemed to make that a safe choice, for example wchar_t. Let's face it, collectively we really screwed this one up. It's the biggest mistake since Microsoft chose the backslash as a path separator in DOS 2.0.

It was actually IBM's fault: they used '/' to denote CLI args in the apps they wrote for DOS 1.0, which didn't have any concept of directories. From Larry Osterman's blog [1]: > Here's a little known secret about MS-DOS. The DOS developers weren't particularly happy about this state of affairs - heck, they all used Xenix machines for email and stuff, so they were familiar with the *nix command semantics. So they code…

Larry has that wrong, IBM wasn't to blame.

IBM licensed DOS from Microsoft. Microsoft bought DOS from Seattle Computer Products QDOS. That software got its command line switches using "/" from CP/M for compatibility reasons; originally, both CP/M and MS-DOS were available for the IBM PC.

CP/M borrowed the convention primarily from RT-11, the OS for the PDP-11, although it wasn't consistently followed there. Programs on RT-11 were responsible for parsing their own command line args, and not all of them used the same convention.

Inside Windows itself, most APIs accept either forward or backward slashes in paths (even both in the same path) without any special incantation. The problem is mainly at the application level where the whole forward/backward slash thing gets messed up because technically you should accept either one from user input and most app code expects one or the other.

Re: Why we can't process Emoji anymore

#62

Post by est »

oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes.

In python:

    len(u'汉字') == 2
    len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options
    len(u'汉字'.encode('utf8')) == 6

Re: Why we can't process Emoji anymore

#63

Post by gsnedders »

othermaciej wrote: post #56
ender7 wrote: post #19

Apropos: http://mathiasbynens.be/notes/javascript-encoding TL;DR: - Javascript engines are free to internally represent strings as either UCS-2 or UTF-16. Engines that choose to go USC-2 tend to replace all glyphs outside of the BMP with the replacement char (U+FFFD). Firefox, IE, Opera, and Safari all do this (with some inconsistencies). - However , from the point of view of the actual JS code that gets executed, st…

Safari uses UTF-16, not UCS-2. I believe this is true of other browsers as well. Otherwise this would render the replacement char, but it doesn't, it renders correctly: javascript:var x = '𝌆';document.write(x);

Well, a JS string is just a series of UTF-16 code-units (per ES5, there is no impl choice here), so there isn't really any encoding pre-se (and isn't necessarily a UTF-16 string, per the spec's definition thereof, as lone surrogates are valid). The fact that that works is more a testament to the the DOM being UTF-16 than JS.

(On the other hand, I'm sure you knew that. But probably there are people reading your comment who didn't. :))

Re: Why we can't process Emoji anymore

#64

Post by dietrichepp »

est wrote: post #62
oofabz wrote: post #16

This is why UTF-8 is great. If it works for any Unicode character it will work for them all. Surrogate pairs are rare enough that they are poorly tested. With UTF-8, if there are issues with multi-byte characters, they are obvious enough to get fixed. UTF-16 is not a very good encoding. It only exists for legacy reasons. It has the same major drawback as UTF-8 (variable-length encoding) but none of the benefits (ASCI…

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing?

UTF-16 solves problems that don't exist.

(Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

Re: Why we can't process Emoji anymore

#65

Post by est »

driverdan wrote: post #4

If you search for V8 UCS-2 you'll find a lot of discussion on this issue dating back at least a few years. There are ways to work around V8's lack of support for surrogate pairs. See this V8 issue for ideas: https://code.google.com/p/v8/issues/detail?id=761 My question is why does V8 (or anything else) still use UCS-2?

because counting 2 bytes is much faster for computers than counting vary 1, 2, 3 or even 4 bytes.

Re: Why we can't process Emoji anymore

#66

Post by gsnedders »

driverdan wrote: post #4

If you search for V8 UCS-2 you'll find a lot of discussion on this issue dating back at least a few years. There are ways to work around V8's lack of support for surrogate pairs. See this V8 issue for ideas: https://code.google.com/p/v8/issues/detail?id=761 My question is why does V8 (or anything else) still use UCS-2?

The ES5 spec defines a string as being a series of UTF-16 code-units, which inherently means surrogates show through.

APIs like that tend to be low priority because they aren't used by browsers (which pass everything through as UTF-16 code-units, typically treating them as possibly-valid UTF-16 strings).

Re: Why we can't process Emoji anymore

#67

Post by [deleted] »

dietrichepp wrote: post #64
est wrote: post #62

Earlier quoted context omitted.

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

[deleted]

Re: Why we can't process Emoji anymore

#68

Post by SeoxyS »

dietrichepp wrote: post #64
est wrote: post #62

Earlier quoted context omitted.

UCS2 is better than UTF8 internally because it counts unicode characters faster than UTF8. Every character is juest 2 bytes, instead of 1, 2, 3 or even 4 bytes. In python: len(u'汉字') == 2 len( '汉字') == 4 # or maybe 6, it varies based on console encoding and CPython options len(u'汉字'.encode('utf8')) == 6

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

It's not a common use case, but I've had to do it. Luckily, it's fairly easy:

    int count_multibytes_encountered(char *text, unsigned long len) {
        int count = 0;
        for (int i = 0; i

Re: Why we can't process Emoji anymore

#70

Post by pjscott »

[deleted] wrote: post #67
dietrichepp wrote: post #64

Earlier quoted context omitted.

Why do you want to count Unicode characters? Why do you care if it is fast to do so? Why would you ever need to use character-based string indexing? UTF-16 solves problems that don't exist. (Honestly, I would love it if someone could explain what the purpose of counting characters is, because I don't know why you'd ever do that, except when you're posting to Twitter.)

[deleted]

If you make an API, please don't do this. If you have to include indexes into a Unicode string, then make them indexes into a binary string in a known encoding. This works equally well for all encodings, and won't tempt people to do something as preposterous as using UCS-2.
Post reply on HN