Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node's Unicode Dragon

cirw.in

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Node's Unicode Dragon

#51

Post by lifthrasiir »

rspeer wrote: post #43
est wrote: post #31

Earlier quoted context omitted.

> 50% is just the absolute worst case. Many kinds of textual data include large amounts of code units that fit in one byte in utf-8 For latin alphabets, yes. For CJK, it's really bad. Things get worse if you dealt with non-BMP before, like iOS emoji, which force you to upgrade MySQL to support utf8mb4, which is totally bullshit. (why the hell do people even presume utf8 is max 3 bytes?)

Except most text isn't plain text. HTML pages in CJK are still smaller in UTF-8 than in their respective countries' favorite encodings.

You are tremendously wrong. Almost every legacy CJK encoding encodes a string in the smaller number of bytes than UTF-8 when the string in question has no unsupported characters in it. I have seen lots of (mostly misguided) people who prefer those legacy encodings over UTF-8/16 solely for this reason.

Re: Node's Unicode Dragon

#52

Post by scoopr »

This same problem manifests with Java as well, where some methods that claim to return UTF-8 on closer inspection actually return “modified UTF-8”, which is broken the same way. Notably I ran across this in with JNI function GetStringUTFChars, but may come across in DataOutputStream's writeUTF etc.

Re: Node's Unicode Dragon

#53

Post by derefr »

justin_vanw wrote: post #40

Man, I'm starting to think there is a cult around JSON. If you need to accept arbitrary binary data, JSON is a profoundly bad choice. At a minimum, you would expect them to base64 encode the data and put that into a JSON string. If you are looking at error reports, how is it even remotely acceptable to have them silently modified to include invalid unicode replacement characters? The lesson here isn't some crappy hac…

Any wire-serialization format that wants to send arbitrary data should really have a "raw binary payload" type. XML has CDATA. ASN.1 has bitstrings. BERT has Binaries. But JSON doesn't really have anything like that.

I wonder... at some point, Javascript could get a convenient literal syntax for creating pre-filled ArrayBuffers, which would basically be the format JSON would want to adopt. But would it? Are changes to Javascript literal syntax folded into JSON, or is JSON now its own thing that doesn't track JS any more?

Re: Node's Unicode Dragon

#54

Post by pyre »

millstone wrote: post #50
deathanatos wrote: post #38

Earlier quoted context omitted.

The good point (in my opinion) is not that "ASCII takes 1 byte, BMP takes 2 bytes, everything else 4 bytes", but rather that the exposed API hides this from you, and exposes to you a sequence of code points. This, I hope, will reduce errors, as code points , not code units, is often a better abstraction to be working with. (For some random string processing function.) So far as I know, Haskell is the only other langu…

Code points is a better abstraction than code units, but it's still a piss-poor abstraction. Consider the problem of producing a valid substring from a Unicode string. It's important that you not split surrogate pairs, and it's true working with code points spares you from that particular problem. But it's also important that you not split combining marks, and zero width joiners, and Hangul syllables... (see http://w…

This was my reaction too. It's Unicode all the way down... :)

Re: Node's Unicode Dragon

#55

Post by Dylan16807 »

derefr wrote: post #53
justin_vanw wrote: post #40

Man, I'm starting to think there is a cult around JSON. If you need to accept arbitrary binary data, JSON is a profoundly bad choice. At a minimum, you would expect them to base64 encode the data and put that into a JSON string. If you are looking at error reports, how is it even remotely acceptable to have them silently modified to include invalid unicode replacement characters? The lesson here isn't some crappy hac…

Any wire-serialization format that wants to send arbitrary data should really have a "raw binary payload" type. XML has CDATA. ASN.1 has bitstrings. BERT has Binaries. But JSON doesn't really have anything like that. I wonder... at some point, Javascript could get a convenient literal syntax for creating pre-filled ArrayBuffers, which would basically be the format JSON would want to adopt. But would it? Are changes t…

CDATA disallows null bytes, so it's even worse than non-support: illusory support

XML doesn't even allow escaped null bytes, so you're basically forced to use base64 or weird custom app-internal escapes.

JSON never tracked javascript. It has one version, period. But you could get people to adopt a superset with a new data type, if you kept it simple.

Re: Node's Unicode Dragon

#56

Post by thaumasiotes »

rspeer wrote: post #43
est wrote: post #31

Earlier quoted context omitted.

> 50% is just the absolute worst case. Many kinds of textual data include large amounts of code units that fit in one byte in utf-8 For latin alphabets, yes. For CJK, it's really bad. Things get worse if you dealt with non-BMP before, like iOS emoji, which force you to upgrade MySQL to support utf8mb4, which is totally bullshit. (why the hell do people even presume utf8 is max 3 bytes?)

Except most text isn't plain text. HTML pages in CJK are still smaller in UTF-8 than in their respective countries' favorite encodings.

well.... China's favorite encoding is GB, which encodes ascii values as one byte and chinese characters as two bytes. It's hard to see how UTF-8 (one byte for ascii, three for characters) would beat that, on the assumption that nearly 100% of what a chinese website would want to transmit is either ascii (where UTF-8 is equivalent to GB) or chinese (where it's inferior).

How do I know GB is preferred? I'm going off of three things:

- According to wikipedia (http://en.wikipedia.org/wiki/GB18030), software sold in China is legally required to support it.

- I was once given a chinese ebook, which I had to figure out was in GB before I could read it. (And now, I know about chardet!)

- I worked with a chinese programmer who accidentally committed files in GB, even though they were supposed to be in UTF-8.

And since the latest GB can in fact represent any unicode point, it's hard to see why it wouldn't be preferred indefinitely.

Re: Node's Unicode Dragon

#57

Post by bsaul »

Reminds me of a previous discussion about Go being more "mature" than node.js, where i said having someone like Pike on board gives you more than 30 years of "maturity". I'm pretty sure you wouldn't find those leaky UTF encoding handling in Go.

Re: Node's Unicode Dragon

#58

Post by ygra »

bsaul wrote: post #57

Reminds me of a previous discussion about Go being more "mature" than node.js, where i said having someone like Pike on board gives you more than 30 years of "maturity". I'm pretty sure you wouldn't find those leaky UTF encoding handling in Go.

Well, Node builds atop an established language, while Go is a new development. It's probably easier to build sane Unicode semantics into a new language than to change the JS spec.

Re: Node's Unicode Dragon

#59

Post by ygra »

Dylan16807 wrote: post #55
derefr wrote: post #53

Earlier quoted context omitted.

Any wire-serialization format that wants to send arbitrary data should really have a "raw binary payload" type. XML has CDATA. ASN.1 has bitstrings. BERT has Binaries. But JSON doesn't really have anything like that. I wonder... at some point, Javascript could get a convenient literal syntax for creating pre-filled ArrayBuffers, which would basically be the format JSON would want to adopt. But would it? Are changes t…

CDATA disallows null bytes, so it's even worse than non-support: illusory support XML doesn't even allow escaped null bytes, so you're basically forced to use base64 or weird custom app-internal escapes. JSON never tracked javascript. It has one version, period. But you could get people to adopt a superset with a new data type, if you kept it simple .

Isn't CDATA character data anyway and thus not even binary but in the document's character set? Which makes it a poor choice for binary data even without taking into account that XML forbids certain characters.

As for binary data in web services ... isn't it easier to just use Content-Type for that and use the appropriate type for the payload? That wouldn't require a textual data format that can contain arbitrary binary data.

Re: Node's Unicode Dragon

#60

Post by pjscott »

bsaul wrote: post #57

Reminds me of a previous discussion about Go being more "mature" than node.js, where i said having someone like Pike on board gives you more than 30 years of "maturity". I'm pretty sure you wouldn't find those leaky UTF encoding handling in Go.

Since Rob Pike and Ken Thompson are the guys who came up with UTF-8, you'd expect them to write decent Unicode encoding for Go. It would be surprising if they didn't.
Post reply on HN