Node's Unicode Dragon

#1

Post by foobar2k » Mon, Sep 09, 2013, 8:21 PM UTC

Re: Node's Unicode Dragon

#2

Post by baddox » Mon, Sep 09, 2013, 9:13 PM UTC

Despite that being a rather interesting technical article, I am upset that my expectation of an actual Unicode depiction of a dragon was not met.

Re: Node's Unicode Dragon

#3

Post by nonchalance » Mon, Sep 09, 2013, 9:16 PM UTC

String encoding in general is a mess. Wait till you get to code pages. Incidentally, the largest JS script I've ever seen pertained to encoding and decoding characters under various codepages: https://raw.github.com/Niggler/js-codepage/master/cptable.js [github complains "(Sorry about that, but we can't show files that are this big right now.)"]

Re: Node's Unicode Dragon

#4

Post by shawnz » Mon, Sep 09, 2013, 9:19 PM UTC

> Unfortunately for us, Javascript has never been updated to support UTF-16. Instead it continues to treat strings as UCS-2.

So really, they were parsing the JSON as if it were UTF-16, but really it was UCS-2. How is that an error in Node?

Re: Node's Unicode Dragon

#5

Post by greenyoda » Mon, Sep 09, 2013, 9:20 PM UTC

baddox wrote: post #2

Despite that being a rather interesting technical article, I am upset that my expectation of an actual Unicode depiction of a dragon was not met.

There is actually a Unicode dragon character at code point U+1F409:

http://www.fileformat.info/info/unicode/char/1f409/index.htm

Also, since any ASCII dragon is also a valid Unicode dragon (in UTF-8, at least), the following might satisfy your needs:

http://www.dougsartgallery.com/ascii-art-dragon.html

Re: Node's Unicode Dragon

#6

Post by cirwin » Mon, Sep 09, 2013, 9:23 PM UTC

baddox wrote: post #2

Despite that being a rather interesting technical article, I am upset that my expectation of an actual Unicode depiction of a dragon was not met.

🐉

To see this dragon, either:

1. Use Safari or Firefox on OS X. 2. Install custom fonts for Linux or Windows. 3. Install https://chrome.google.com/webstore/detail/chromoji-emoji-for... for Chrome

Re: Node's Unicode Dragon

#7

Post by justincormack » Mon, Sep 09, 2013, 9:29 PM UTC

shawnz wrote: post #4

> Unfortunately for us, Javascript has never been updated to support UTF-16. Instead it continues to treat strings as UCS-2. So really, they were parsing the JSON as if it were UTF-16, but really it was UCS-2. How is that an error in Node?

JSON is defined as UTF8, 16 or 32 [1]. The escaped characters are UTF-16 not UCS2. It is unfortunate of JavaScript can't parse it correctly!

[1] http://www.ietf.org/rfc/rfc4627.txt

Re: Node's Unicode Dragon

#8

Post by state » Mon, Sep 09, 2013, 9:29 PM UTC

Whew. This explains a bug from six months ago that drove me up the wall. I could never figure it out.

Re: Node's Unicode Dragon

#9

Post by dsj36 » Mon, Sep 09, 2013, 9:31 PM UTC

how did the error JSON include the undecodable bytes? JSON strings are all unicode sequences, so there would have had to be some way that the raw bytes were mapped into codepoints.

on the other hand, if the offending bytes were blindly substituted into the JSON, then it's not surprising that there were decoding issues down the line...

Re: Node's Unicode Dragon

#10

Post by kansface » Mon, Sep 09, 2013, 9:32 PM UTC

shawnz wrote: post #4

> Unfortunately for us, Javascript has never been updated to support UTF-16. Instead it continues to treat strings as UCS-2. So really, they were parsing the JSON as if it were UTF-16, but really it was UCS-2. How is that an error in Node?

They wanted to parse some bytes as utf-16, but are unable to do so because V8 only understands ucs2 (with invalid surrogate pairs). This is a major problem with node- ie, it happily produces/consumes invalid unicode encoded strings.