Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node's Unicode Dragon

cirw.in

Post reply on HN
Print view View original

61–66 of 66 posts

Re: Node's Unicode Dragon

#61

Post by derefr »

ygra wrote: post #59
Dylan16807 wrote: post #55

Earlier quoted context omitted.

CDATA disallows null bytes, so it's even worse than non-support: illusory support XML doesn't even allow escaped null bytes, so you're basically forced to use base64 or weird custom app-internal escapes. JSON never tracked javascript. It has one version, period. But you could get people to adopt a superset with a new data type, if you kept it simple .

Isn't CDATA character data anyway and thus not even binary but in the document's character set? Which makes it a poor choice for binary data even without taking into account that XML forbids certain characters. As for binary data in web services ... isn't it easier to just use Content-Type for that and use the appropriate type for the payload? That wouldn't require a textual data format that can contain arbitrary bin…

That presumes you want to send a lot of binary data. Sometimes you just have five bytes or so. (E.g., as in the article, an un-decoded string.)

(Still, you're right, I admit to having been able to avoid any work painful enough to teach me XML arcana. I was actually thinking of one of the many variants of "Binary XML" I had read about recently, and assumed the typing was bijective to XML's own types. In other news, BSON of all things has a raw-binary type.)

Re: Node's Unicode Dragon

#62

Post by twoodfin »

jlarocco wrote: post #21
dsj36 wrote: post #9

how did the error JSON include the undecodable bytes? JSON strings are all unicode sequences, so there would have had to be some way that the raw bytes were mapped into codepoints. on the other hand, if the offending bytes were blindly substituted into the JSON, then it's not surprising that there were decoding issues down the line...

From the article: > The exceptions that were crashing us were caused by people using String.prototype.substr. That function works perfectly on strings that only contain Unicode 1.0 data, but as soon as you're storing UTF-16 in your UCS-2 string there's a possibility that when you take a slice you'll split a valid surrogate pair into two invalid lonely surrogates. To me, it seems like it'd be nearly impossible for som…

These kinds of isolated surrogate pairs are pretty easy to create if you're doing the right kind of processing on the right kind of data.

Suppose you receive a long piece of text wrapped in JSON, unpack it into a JS String, then start processing it in fixed size chunks. If your source text contains any significant percentage of surrogate pair-represented characters, you'll eventually break one.

Re: Node's Unicode Dragon

#63

Post by acdha »

est wrote: post #37
acdha wrote: post #36

Earlier quoted context omitted.

The difference is internal vs. external: if you use UTF-8 things simply work better any time you exchange data. If you had lots of high code points (>\U+8000) you would get better results in most cases using a full compression system rather than playing with 2-byte encoding hacks, which is really all you're talking about in both cases.

I totally agree that UTF-8 is pretty good at exchanging data because UTF8 is better than UCS-* and other UTF-* overall, and because everyone is (and should be) using it. But you know, there are other cases besides exchanging. Like I said, if your text data is mainly latin you are good, but not so good if you are stuck with non-latin BMPs.

I'm defining “exchange” as not just the outside world but also among components in your own architecture. I've flushed out enough problems with the UCS-2 / UTF-16 breakage that I've become somewhat sold on the unambiguous nature of UTF-8.

For the classic double-byte languages, what's your total data size after compression? e.g. in the case of full-text search, enabling compression has been enough of a win that the 2/3-byte expansion hasn't been a challenge, particularly since the biggest UTF-8 drawback (inability to predict total byte string length) isn't an issue when working with a data structure which records the length of each record.

Re: Node's Unicode Dragon

#64

Post by gnaritas »

daxelrod wrote: post #41
gnaritas wrote: post #18

Earlier quoted context omitted.

Not really as JSON is not valid JavaScript and requires its own parser. It's based on JavaScript, but it is not JavaScript.

I was skeptical, but I did some searching, and you appear to be right! The difference seems to come down to string handling: http://timelessrepo.com/json-isnt-a-javascript-subset

Ha, same article where I first learned this.

Re: Node's Unicode Dragon

#65

Post by cmccabe »

deathanatos wrote: post #38
est wrote: post #26

Earlier quoted context omitted.

Because some of us are pissed that some BMP characters takes 3 bytes in UTF8, that's 50% more waste of storage space and 50% more time to read/write. I like the design of Python 3.3 encoding. ASCII takes 1 byte, BMP takes 2 bytes, everything else 4 bytes. http://www.python.org/dev/peps/pep-0393/

The good point (in my opinion) is not that "ASCII takes 1 byte, BMP takes 2 bytes, everything else 4 bytes", but rather that the exposed API hides this from you, and exposes to you a sequence of code points. This, I hope, will reduce errors, as code points , not code units, is often a better abstraction to be working with. (For some random string processing function.) So far as I know, Haskell is the only other langu…

Go allows you to iterate over a string as a series of code points.

Re: Node's Unicode Dragon

#66

Post by rspeer »

thaumasiotes wrote: post #56
rspeer wrote: post #43

Earlier quoted context omitted.

Except most text isn't plain text. HTML pages in CJK are still smaller in UTF-8 than in their respective countries' favorite encodings.

well.... China's favorite encoding is GB, which encodes ascii values as one byte and chinese characters as two bytes. It's hard to see how UTF-8 (one byte for ascii, three for characters) would beat that, on the assumption that nearly 100% of what a chinese website would want to transmit is either ascii (where UTF-8 is equivalent to GB) or chinese (where it's inferior). How do I know GB is preferred? I'm going off of…

Okay, you are right. The comparison I was thinking of was actually about UTF-16, and of course that's not actually preferred to GB or Shift-JIS or whatever.
Post reply on HN