Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RJSON: compress JSON to JSON

cliws.com

Post reply on HN
Print view View original

11–20 of 45 posts

Re: RJSON: compress JSON to JSON

#11

Post by ZenPsycho »

whalesalad wrote: post #9
ZenPsycho wrote: post #2

I see a pretty major problem with this. It seems to depend on the order of key value pairs in object literals being defined. The order is not defined by the JSON or the ecmascript standards. So you can't really depend on the order of keys in a json object, unless you explicitly define some order (like alphabetical, for instance). I like the basic concept of compressing json to json but this is not a particularly good…

I saw the same thing right away. Seems like the first item in the array would have to be used as a hash of key/index pairs to be used for the following items. I guess this would only be useful with large sets of data, the examples makes them look kind of silly =) So this: "users": [ {"first": "Homer", "last": "Simpson"}, {"first": "Hank", "last": "Hill"}, {"first": "Peter", "last": "Griffin"} ], Becomes: "users": [ […

if you can infer a schema, I'd almost prefer a column oriented arrangement.

   "users": 
        {
         "first":["Homer","Hank","Peter"], 
         "last":["Simpson","Hill","Griffin"]
         },
then the information about the original structure can be restored by a set of object paths which need to be "rotated" from column to row orientation. ["users"]

saving a few characters in the process. though I see that the advantage of this system is supposedly that it can handle any sort of shape of data, not just ones with a fixed schema. I've been trying to figure out how trang [http://www.thaiopensource.com/relaxng/trang.html] does its schema inference trick. (it turns a set of xml files into a relaxNG schema). If you have a schema for a JSON file, it's knowledge you can apply to algorithmically creating really efficient transformations.

Re: RJSON: compress JSON to JSON

#12

Post by zippie »

Would like to emphasize that this is only really useful in environments where gzip is not available (as the OP notes)...some tests using the demo JSON (minified):

test.json = 285 bytes

test.rjson = 233 bytes (18%)

test.json.gz = 205 bytes (27%)

If you are able to bundle a RJSON parser, why not just bundle an existing, well understood/tested compression scheme such as http://stuartk.com/jszip/ or https://github.com/olle/lz77-kit/blob/master/src/main/js/lz7... instead?

Re: RJSON: compress JSON to JSON

#14

Post by microtonal »

whalesalad wrote: post #9
ZenPsycho wrote: post #2

I see a pretty major problem with this. It seems to depend on the order of key value pairs in object literals being defined. The order is not defined by the JSON or the ecmascript standards. So you can't really depend on the order of keys in a json object, unless you explicitly define some order (like alphabetical, for instance). I like the basic concept of compressing json to json but this is not a particularly good…

I saw the same thing right away. Seems like the first item in the array would have to be used as a hash of key/index pairs to be used for the following items. I guess this would only be useful with large sets of data, the examples makes them look kind of silly =) So this: "users": [ {"first": "Homer", "last": "Simpson"}, {"first": "Hank", "last": "Hill"}, {"first": "Peter", "last": "Griffin"} ], Becomes: "users": [ […

Doesn't this introduce ambiguity? How do you represent list of 'tuple lists'?

Re: RJSON: compress JSON to JSON

#15

Post by ZenPsycho »

microtonal wrote: post #14
whalesalad wrote: post #9

Earlier quoted context omitted.

I saw the same thing right away. Seems like the first item in the array would have to be used as a hash of key/index pairs to be used for the following items. I guess this would only be useful with large sets of data, the examples makes them look kind of silly =) So this: "users": [ {"first": "Homer", "last": "Simpson"}, {"first": "Hank", "last": "Hill"}, {"first": "Peter", "last": "Griffin"} ], Becomes: "users": [ […

Doesn't this introduce ambiguity? How do you represent list of 'tuple lists'?

you could steal the method rjson uses and do this

   "users": [
        [3, "first", "last"],
        [3, "Homer", "Simpson"],
        [3, "Hank", "Hill"],
        [3, "Peter", "Griffin"]
    ],

Re: RJSON: compress JSON to JSON

#17

Post by microtonal »

zippie wrote: post #12

Would like to emphasize that this is only really useful in environments where gzip is not available (as the OP notes)...some tests using the demo JSON (minified): test.json = 285 bytes test.rjson = 233 bytes (18%) test.json.gz = 205 bytes (27%) If you are able to bundle a RJSON parser, why not just bundle an existing, well understood/tested compression scheme such as http://stuartk.com/jszip/ or https://github.com/ol…

And if you used gzip on a file, is has some overhead (the 10-byte gzip header) and a freshly initialized deflate state. Usually, compression improves when more data is seen, since the dynamic Huffman tree improves and there are more blocks for LZ77 to backreference.

Re: RJSON: compress JSON to JSON

#18

Post by ZenPsycho »

zippie wrote: post #12

Would like to emphasize that this is only really useful in environments where gzip is not available (as the OP notes)...some tests using the demo JSON (minified): test.json = 285 bytes test.rjson = 233 bytes (18%) test.json.gz = 205 bytes (27%) If you are able to bundle a RJSON parser, why not just bundle an existing, well understood/tested compression scheme such as http://stuartk.com/jszip/ or https://github.com/ol…

An arithmetic coding scheme which has a model based on the probabilities found in JSON abstract syntax trees would significantly improve on most typically used generic compression schemes. Arithmetic coding schemes have largely been avoided thus far due to patents which have recently expired, if I remember correctly.

using the order 2 precise model on this page I get 190 bytes-- and that is still a generic non-json model. http://nerget.com/compression/

Re: RJSON: compress JSON to JSON

#19

Post by lsb »

uncle-ezra wrote: post #16

If your HTTP connection is going to be gzip compressed, then manual compression of this kind is not guaranteed to reduce the size of the final result, and may in fact hurt it.

If your HTTP connection is going to be gzip compressed, then this will only give you monetary savings to compensate for the additional code complexity if you have medium to big data, at which point storing it in JSON is a questionable-at-best proposition.

Re: RJSON: compress JSON to JSON

#20

Post by zippie »

ZenPsycho wrote: post #18
zippie wrote: post #12

Would like to emphasize that this is only really useful in environments where gzip is not available (as the OP notes)...some tests using the demo JSON (minified): test.json = 285 bytes test.rjson = 233 bytes (18%) test.json.gz = 205 bytes (27%) If you are able to bundle a RJSON parser, why not just bundle an existing, well understood/tested compression scheme such as http://stuartk.com/jszip/ or https://github.com/ol…

An arithmetic coding scheme which has a model based on the probabilities found in JSON abstract syntax trees would significantly improve on most typically used generic compression schemes. Arithmetic coding schemes have largely been avoided thus far due to patents which have recently expired, if I remember correctly. using the order 2 precise model on this page I get 190 bytes-- and that is still a generic non-json m…

This - JSON specific compression schemes aren't going to yield gains over AST friendly schemes unless the JSON serialization specification changes significantly.

Along these lines - shipping a schema with the data payload is avro-like ... which is also questionable in terms of efficiency when compared with gzip/LZO.

Post reply on HN