Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZSON, PostgreSQL extension for compressing JSONB

postgresql.org

Post reply on HN
Print view View original

1–10 of 14 posts

Re: ZSON, PostgreSQL extension for compressing JSONB

#3

Post by vog »

exo762 wrote: post #2

It would be nice to see a comparison to a general compression algorithm - e.g. deflate.

There is a comparison table at the end of the project's README (https://github.com/afiskon/zson). However, the table columns are not explained very well. It is not 100% clear to me whether "before" means "uncompressed" or "PGLZ compressed".

  Compression ratio could be different depending on documents,
  database schema, number of rows, etc. But in general ZSON
  compression is much better than build-in PostgreSQL
  compression (PGLZ):

     before   |   after    |      ratio       
  ------------+------------+------------------
   3961880576 | 1638834176 | 0.41365057440843
  (1 row)
  
     before   |   after    |       ratio       
  ------------+------------+-------------------
   8058904576 | 4916436992 | 0.610062688500061
  (1 row)
  
     before    |   after    |       ratio       
  -------------+------------+-------------------
   14204420096 | 9832841216 | 0.692238130775149

Re: ZSON, PostgreSQL extension for compressing JSONB

#4

Post by afiskon »

vog wrote: post #3
exo762 wrote: post #2

It would be nice to see a comparison to a general compression algorithm - e.g. deflate.

There is a comparison table at the end of the project's README ( https://github.com/afiskon/zson ). However, the table columns are not explained very well. It is not 100% clear to me whether "before" means "uncompressed" or "PGLZ compressed". Compression ratio could be different depending on documents, database schema, number of rows, etc. But in general ZSON compression is much better than build-in PostgreSQL compre…

Frankly I don't remember all details since I did this benchmark in February. IIRC its "ZSON + PGLZ" vs "JSONB + PGLZ".

Please note that everything depends very much on your data. PostgreSQL is smart about what to compress and what not. In general it could be all combinations of "ZSON +/- PGLZ" vs "JSONB +/- PGLZ".

Don't believe any benchmark I or anyone else did. Re-check everything on your data, configuration, hardware, workload, etc.

Re: ZSON, PostgreSQL extension for compressing JSONB

#7

Post by afiskon »

brianolson wrote: post #6

Another way to make 'JSON' smaller is to instead use 'CBOR', schema compatible 'concise binary object representation'. (See IETF RFC 7049 or http://cbor.io/ ) CBOR encodes and decodes faster too. Or use the 'snappy' compressor.

I'm afraid Snappy will not help, at least a lot. PostgreSQL already has a build-in compression (PGLZ). I tested various different algorithms before this shared-dictionary-idea - lz4, bzip and others. Some compress a bit better, other a bit faster, but in general result is almost the same.

Re: ZSON, PostgreSQL extension for compressing JSONB

#9

Post by tveita »

afiskon wrote: post #7
brianolson wrote: post #6

Another way to make 'JSON' smaller is to instead use 'CBOR', schema compatible 'concise binary object representation'. (See IETF RFC 7049 or http://cbor.io/ ) CBOR encodes and decodes faster too. Or use the 'snappy' compressor.

I'm afraid Snappy will not help, at least a lot. PostgreSQL already has a build-in compression (PGLZ). I tested various different algorithms before this shared-dictionary-idea - lz4, bzip and others. Some compress a bit better, other a bit faster, but in general result is almost the same.

Did you test e.g. LZ4 with a prebuilt dictionary? With a good way to find substrings for the dictionary it might generalize well to other kinds of data.
Post reply on HN