Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RJSON: compress JSON to JSON

cliws.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: RJSON: compress JSON to JSON

#21

Post by snorkel »

whalesalad wrote: post #9
ZenPsycho wrote: post #2

I see a pretty major problem with this. It seems to depend on the order of key value pairs in object literals being defined. The order is not defined by the JSON or the ecmascript standards. So you can't really depend on the order of keys in a json object, unless you explicitly define some order (like alphabetical, for instance). I like the basic concept of compressing json to json but this is not a particularly good…

I saw the same thing right away. Seems like the first item in the array would have to be used as a hash of key/index pairs to be used for the following items. I guess this would only be useful with large sets of data, the examples makes them look kind of silly =) So this: "users": [ {"first": "Homer", "last": "Simpson"}, {"first": "Hank", "last": "Hill"}, {"first": "Peter", "last": "Griffin"} ], Becomes: "users": [ […

Prefer this approach. It's more readable and less fragile.

Re: RJSON: compress JSON to JSON

#22

Post by dogada »

ZenPsycho wrote: post #11
whalesalad wrote: post #9

Earlier quoted context omitted.

I saw the same thing right away. Seems like the first item in the array would have to be used as a hash of key/index pairs to be used for the following items. I guess this would only be useful with large sets of data, the examples makes them look kind of silly =) So this: "users": [ {"first": "Homer", "last": "Simpson"}, {"first": "Hank", "last": "Hill"}, {"first": "Peter", "last": "Griffin"} ], Becomes: "users": [ […

if you can infer a schema, I'd almost prefer a column oriented arrangement. "users": { "first":["Homer","Hank","Peter"], "last":["Simpson","Hill","Griffin"] }, then the information about the original structure can be restored by a set of object paths which need to be "rotated" from column to row orientation. ["users"] saving a few characters in the process. though I see that the advantage of this system is supposedly…

Exactly, ZenPsycho, the purpose of RJSON is to convert data with dynamic schema. Fields with default values are omitted often, for example if most of data have 'private' property set to False, it have sense to output it only for 1% of objects with 'private' set to True. This issue is addressed in RJSON.

Re: RJSON: compress JSON to JSON

#23

Post by dogada »

microtonal wrote: post #14
whalesalad wrote: post #9

Earlier quoted context omitted.

I saw the same thing right away. Seems like the first item in the array would have to be used as a hash of key/index pairs to be used for the following items. I guess this would only be useful with large sets of data, the examples makes them look kind of silly =) So this: "users": [ {"first": "Homer", "last": "Simpson"}, {"first": "Hank", "last": "Hill"}, {"first": "Peter", "last": "Griffin"} ], Becomes: "users": [ […

Doesn't this introduce ambiguity? How do you represent list of 'tuple lists'?

You can see it in the unit tests: https://github.com/dogada/RJSON/blob/master/test/tests.js#L4...

Re: RJSON: compress JSON to JSON

#24

Post by ZenPsycho »

zippie wrote: post #20
ZenPsycho wrote: post #18

Earlier quoted context omitted.

An arithmetic coding scheme which has a model based on the probabilities found in JSON abstract syntax trees would significantly improve on most typically used generic compression schemes. Arithmetic coding schemes have largely been avoided thus far due to patents which have recently expired, if I remember correctly. using the order 2 precise model on this page I get 190 bytes-- and that is still a generic non-json m…

This - JSON specific compression schemes aren't going to yield gains over AST friendly schemes unless the JSON serialization specification changes significantly. Along these lines - shipping a schema with the data payload is avro-like ... which is also questionable in terms of efficiency when compared with gzip/LZO.

hey look, I found this http://research.microsoft.com/en-us/projects/jszap/

Re: RJSON: compress JSON to JSON

#25

Post by dogada »

asdfaoeu wrote: post #6
perfunctory wrote: post #3

How would it compress and then decompress the following document? {"users": [ {"first": "Homer", "last": "Simpson"}, {"first": "Hank", "last": "Hill"}, [2, "Peter", "Griffin"] }

{ "users": [ { "first": "Homer", "last": "Simpson" }, [ 2, "Hank", "Hill" ], [ 0, 2, "Peter", "Griffin" ] ] } There was a tester linked from the site. Looks like he added the [0, to handle that case.

Yeah, for arrays with first numeric value is reserved schema with index 0. You can see more examples in the unit tests: https://github.com/dogada/RJSON/blob/master/test/tests.js

Re: RJSON: compress JSON to JSON

#27

Post by zippie »

ZenPsycho wrote: post #24
zippie wrote: post #20

Earlier quoted context omitted.

This - JSON specific compression schemes aren't going to yield gains over AST friendly schemes unless the JSON serialization specification changes significantly. Along these lines - shipping a schema with the data payload is avro-like ... which is also questionable in terms of efficiency when compared with gzip/LZO.

hey look, I found this http://research.microsoft.com/en-us/projects/jszap/

They are using gzip compression level 1. Bogus.

Re: RJSON: compress JSON to JSON

#28

Post by jerf »

dogada wrote: post #25
asdfaoeu wrote: post #6

Earlier quoted context omitted.

{ "users": [ { "first": "Homer", "last": "Simpson" }, [ 2, "Hank", "Hill" ], [ 0, 2, "Peter", "Griffin" ] ] } There was a tester linked from the site. Looks like he added the [0, to handle that case.

Yeah, for arrays with first numeric value is reserved schema with index 0. You can see more examples in the unit tests: https://github.com/dogada/RJSON/blob/master/test/tests.js

Unit test suggestion: testPackAndUnpack should also check double-compress and double-decompress for being identical, it'll tend to find any remaining such issues, if any.

Re: RJSON: compress JSON to JSON

#30

Post by Too »

Good idea but i think for those extreme cases where you really have use for this you might as well go with a specialized protocol, which can be based on json if you need it.

I had to do this for an application which streamed several hundreds of data points per second to a browser-client. Both data-size on the network and parsing time in the browser was my biggest issues. Since it was a html-app i had to use either JSON.parse or custom parsing written in javascript, the second option being too slow to be viable. I ended up with something based on almost only json-arrays and then the client would know what the different items in the array meant. With his example it would only look something this: [7, ["programming", "javascript"],["Peter", "Griffin", "Homer", "Simpson", "Hank", "Hill"]]

So in other words it's just enough overhead to make it parsable by json.parse but otherwise you only transfer the actual data.

Note that I wouldn't recommend this option unless you really hit a wall where you realize that this in fact is a bottleneck.

Post reply on HN