RJSON: compress JSON to JSON • Hacker News Board

Re: RJSON: compress JSON to JSON

#31

Post by jimktrains2 » Tue, Jun 05, 2012, 3:10 PM UTC

mhansen wrote: post #10

jimktrains2 wrote: post #8

> reduce JSON data size and network traffic when gzip isn't available. For example, in-browser 3D-modeling tools like Mydeco 3D-planner may process and send to server megabytes of JSON-data; `Content-encoding: gzip` anyone?

I don't believe that browsers let uploads be gzipped, as it can't be sure that the server supports gzipped requests.

Good call. I would think there should be a way to specify it... Something to look into I guess.

EDIT: or as a comment above states, compress (gzip/deflate) it yourself. Not the most elegant, but if space is an issue.

Re: RJSON: compress JSON to JSON

#32

Post by gfosco » Tue, Jun 05, 2012, 3:16 PM UTC

I wondered if it was April Fools day... Really, JSON is lightweight enough that it doesn't need compressing. We're not dealing with XML here! This just adds another point of failure.

Re: RJSON: compress JSON to JSON

#33

Post by gcb » Tue, Jun 05, 2012, 3:46 PM UTC

It's all fun until someone loses an eye.

What if i look at your API output and assume it's json (only got unique items) but it's rjson? Or whatever?

The most important thing when adding another layer to a protocol is identification.

So, please, put the whole altered object into a rjson root node so it's clear what we're dealing with.

Re: RJSON: compress JSON to JSON

#34

Post by dogada » Tue, Jun 05, 2012, 4:12 PM UTC

ZenPsycho wrote: post #2

I see a pretty major problem with this. It seems to depend on the order of key value pairs in object literals being defined. The order is not defined by the JSON or the ecmascript standards. So you can't really depend on the order of keys in a json object, unless you explicitly define some order (like alphabetical, for instance). I like the basic concept of compressing json to json but this is not a particularly good…

Thanks ZenPsycho, I added sorting of object schema keys to fix this issue: https://github.com/dogada/RJSON/commit/a27c8927cd0c2d7d151e2...

Re: RJSON: compress JSON to JSON

#35

Post by chubbard » Tue, Jun 05, 2012, 4:17 PM UTC

If you just use GZIP, supported by most browsers, it compresses without the need of special software on the client side to rebuild the original JSON response. And does a better job at compression I bet. I thought of doing something like this to rebuild cycles in object graphs, but I didn't because it requires special parsing logic for the client to use.

Re: RJSON: compress JSON to JSON

#36

Post by dogada » Tue, Jun 05, 2012, 4:32 PM UTC

Too wrote: post #30

Good idea but i think for those extreme cases where you really have use for this you might as well go with a specialized protocol, which can be based on json if you need it. I had to do this for an application which streamed several hundreds of data points per second to a browser-client. Both data-size on the network and parsing time in the browser was my biggest issues. Since it was a html-app i had to use either JS…

RJSON not only compresses data of any structure but keeps it in JSON format that is huge advantage over gzip/7z and other binary compression algorithms. You can compress 1M of JSON to 300Kb of gzipped JSON but then you will to parse 1Mb of JSON on client anyway. With RJSON you can compress 1Mb of JSON to 500Kb of RJSON, then gzip it down to 200Kb and parse only 500Kb of JSON. Then you can analyze some core properties of parsed RJSON document and unpack it on demand. For example you may send to client collection of 20 documents, parse each and show title of each document to the client, then fully unpack only selected by client document.

I agree with Too that specialized protocol will always win, but RJSON IMO decreases structural entropy almost to 0 without need to debugging own protocol.

Re: RJSON: compress JSON to JSON

#37

Post by dogada » Tue, Jun 05, 2012, 4:37 PM UTC

ZenPsycho wrote: post #5

[deleted] wrote: post #4

Earlier quoted context omitted.

[deleted]

that is only enough to make it seem like it works okay. But as I said, there's nothing in the JSON spec that obligates any intervening party to preserve the order of the keys in the object.

It doesn't matter. Even if order of keys will be changed somewhere in the middle during JSON.strigify, as schema id will be always used keys sorted alphabetically and object values are always stored in the same order as keys sorted: https://github.com/dogada/RJSON/blob/master/rjson.js#L213

Re: RJSON: compress JSON to JSON

#38

Post by orthecreedence » Tue, Jun 05, 2012, 4:56 PM UTC

gcb wrote: post #33

It's all fun until someone loses an eye. What if i look at your API output and assume it's json (only got unique items) but it's rjson? Or whatever? The most important thing when adding another layer to a protocol is identification. So, please, put the whole altered object into a rjson root node so it's clear what we're dealing with.

Correct me if I'm wrong, but isn't this what Content-type is for?

Re: RJSON: compress JSON to JSON

#39

Post by dogada » Tue, Jun 05, 2012, 4:59 PM UTC

jerf wrote: post #28

dogada wrote: post #25

Earlier quoted context omitted.

Yeah, for arrays with first numeric value is reserved schema with index 0. You can see more examples in the unit tests: https://github.com/dogada/RJSON/blob/master/test/tests.js

Unit test suggestion: testPackAndUnpack should also check double-compress and double-decompress for being identical, it'll tend to find any remaining such issues, if any.

Thanks, jerf, done: https://github.com/dogada/RJSON/commit/0a7fc5dd9e142477ebda9...

Re: RJSON: compress JSON to JSON

#40

Post by dogada » Tue, Jun 05, 2012, 5:06 PM UTC

gcb wrote: post #33

It's all fun until someone loses an eye. What if i look at your API output and assume it's json (only got unique items) but it's rjson? Or whatever? The most important thing when adding another layer to a protocol is identification. So, please, put the whole altered object into a rjson root node so it's clear what we're dealing with.

I agree that identification is important issue, but IMO it's protocol level issue. RJSON is not protocol, it's algorithm. Someone will prefer to wrap all in {"rjson": ...}, someone like {"format": "rjson", "data": ...} and so on. I belive, algorithm itself should create as less limitations as possible.