Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node's Unicode Dragon

cirw.in

Post reply on HN
Print view View original

11–20 of 66 posts

Re: Node's Unicode Dragon

#11

Post by kansface »

justincormack wrote: post #7
shawnz wrote: post #4

> Unfortunately for us, Javascript has never been updated to support UTF-16. Instead it continues to treat strings as UCS-2. So really, they were parsing the JSON as if it were UTF-16, but really it was UCS-2. How is that an error in Node?

JSON is defined as UTF8, 16 or 32 [1]. The escaped characters are UTF-16 not UCS2. It is unfortunate of JavaScript can't parse it correctly! [1] http://www.ietf.org/rfc/rfc4627.txt

This is true of JSON, but its not true of Javascript which gives no fucks about utf16 (or valid surrogate pairs). Its a very strange world where JSON and Javascript have incompatible interpretations of strings.

http://mathiasbynens.be/notes/javascript-encoding

Re: Node's Unicode Dragon

#12

Post by cirwin »

dsj36 wrote: post #9

how did the error JSON include the undecodable bytes? JSON strings are all unicode sequences, so there would have had to be some way that the raw bytes were mapped into codepoints. on the other hand, if the offending bytes were blindly substituted into the JSON, then it's not surprising that there were decoding issues down the line...

In the example I looked at to debug this, the sequence of events was:

1. One of our customer's javascript apps sent a truncated string to their web-server in a JSON payload. This string ended with a leading surrogate (this is another instance of V8 bug discussed in the blog post).

2. Their ruby backend exploded when they tried to use a regular expression on the string (because ruby's regexp library is strict about valid utf-8).

3. The bugsnag exception notifier copied the bytes from the incoming parameter into the JSON exception notification payload (ruby didn't notice because its string library unconditionally believes you if you tell it a string is valid utf8 — another bug :p).

Re: Node's Unicode Dragon

#13

Post by Wilya »

cirwin wrote: post #6
baddox wrote: post #2

Despite that being a rather interesting technical article, I am upset that my expectation of an actual Unicode depiction of a dragon was not met.

🐉 To see this dragon, either: 1. Use Safari or Firefox on OS X. 2. Install custom fonts for Linux or Windows. 3. Install https://chrome.google.com/webstore/detail/chromoji-emoji-for... for Chrome

Next time I have some "Here be dragons" code, I'm going to use this.

Re: Node's Unicode Dragon

#14

Post by sujayakar »

cirwin wrote: post #12
dsj36 wrote: post #9

how did the error JSON include the undecodable bytes? JSON strings are all unicode sequences, so there would have had to be some way that the raw bytes were mapped into codepoints. on the other hand, if the offending bytes were blindly substituted into the JSON, then it's not surprising that there were decoding issues down the line...

In the example I looked at to debug this, the sequence of events was: 1. One of our customer's javascript apps sent a truncated string to their web-server in a JSON payload. This string ended with a leading surrogate (this is another instance of V8 bug discussed in the blog post). 2. Their ruby backend exploded when they tried to use a regular expression on the string (because ruby's regexp library is strict about va…

ah yeah step 3 seems pretty bad -- cool that you found that bug!

Re: Node's Unicode Dragon

#15

Post by pavlov »

cirwin wrote: post #6
baddox wrote: post #2

Despite that being a rather interesting technical article, I am upset that my expectation of an actual Unicode depiction of a dragon was not met.

🐉 To see this dragon, either: 1. Use Safari or Firefox on OS X. 2. Install custom fonts for Linux or Windows. 3. Install https://chrome.google.com/webstore/detail/chromoji-emoji-for... for Chrome

The dragon glyph is rendered correctly in IE10 on Windows 8 without any custom fonts. Hooray for the most underestimated browser ever ;)

Re: Node's Unicode Dragon

#16

Post by jrochkind1 »

The OP describes an environment where data goes from node to Rails.

If you want to check a string for valid encoding and/or replace bad bytes with replacement char on the _ruby_ end... it's not very obvious how you do that with the ruby stdlib api, and it takes a few tricks to do right.

So I wrote a gem for it: https://github.com/jrochkind/ensure_valid_encoding

Re: Node's Unicode Dragon

#17

Post by city41 »

pavlov wrote: post #15
cirwin wrote: post #6

Earlier quoted context omitted.

🐉 To see this dragon, either: 1. Use Safari or Firefox on OS X. 2. Install custom fonts for Linux or Windows. 3. Install https://chrome.google.com/webstore/detail/chromoji-emoji-for... for Chrome

The dragon glyph is rendered correctly in IE10 on Windows 8 without any custom fonts. Hooray for the most underestimated browser ever ;)

Also true of mobile IE10

Re: Node's Unicode Dragon

#18

Post by gnaritas »

kansface wrote: post #11
justincormack wrote: post #7

Earlier quoted context omitted.

JSON is defined as UTF8, 16 or 32 [1]. The escaped characters are UTF-16 not UCS2. It is unfortunate of JavaScript can't parse it correctly! [1] http://www.ietf.org/rfc/rfc4627.txt

This is true of JSON, but its not true of Javascript which gives no fucks about utf16 (or valid surrogate pairs). Its a very strange world where JSON and Javascript have incompatible interpretations of strings. http://mathiasbynens.be/notes/javascript-encoding

Not really as JSON is not valid JavaScript and requires its own parser. It's based on JavaScript, but it is not JavaScript.

Re: Node's Unicode Dragon

#19

Post by stormbrew »

Wish I'd known about this when I was pointing out in another HN thread how utf-16 is a terrible encoding for, among other reasons, pushing the corner case where you find out your encoding/decoding is broken to the very edge of likelihood. It's ridiculous that v8 doesn't properly support utf16, but it's to be expected I suppose.

UTF-8 does not have this problem. That's the way we should be moving.

Re: Node's Unicode Dragon

#20

Post by drivebyacct2 »

cirwin wrote: post #6
baddox wrote: post #2

Despite that being a rather interesting technical article, I am upset that my expectation of an actual Unicode depiction of a dragon was not met.

🐉 To see this dragon, either: 1. Use Safari or Firefox on OS X. 2. Install custom fonts for Linux or Windows. 3. Install https://chrome.google.com/webstore/detail/chromoji-emoji-for... for Chrome

Works fine in IE11.0.9600.16384 as well as Firefox{,-ux,-nightly} in Windows and every browser in my Arch machine.
Post reply on HN