Citus 10 brings columnar compression to Postgres
citusdata.com
Citus 10 brings columnar compression to Postgres
1–10 of 62 posts
Re: Citus 10 brings columnar compression to Postgres
#2I'm curious to see how this compares in real life to TimescaleDB hypertables with compression - which to me, reads as much the same thing. I'm wondering if Citus is bringing a lower level implementation of idea possibly?
Re: Citus 10 brings columnar compression to Postgres
#3I'm curious to see how this compares in real life to TimescaleDB hypertables with compression - which to me, reads as much the same thing. I'm wondering if Citus is bringing a lower level implementation of idea possibly?
This reads to me more like parquet.
Re: Citus 10 brings columnar compression to Postgres
#4I'm curious to see how this compares in real life to TimescaleDB hypertables with compression - which to me, reads as much the same thing. I'm wondering if Citus is bringing a lower level implementation of idea possibly?
Came here to say this - I was looking to see how compression compared to timescale’s stated 91% compression.
https://docs.timescale.com/latest/using-timescaledb/compress...
Re: Citus 10 brings columnar compression to Postgres
#5One of the gotchas of columnar storage (coming from Redshift) is that you lose all of the compression benefits if you have just one column that’s fat or hard to compress.
In Redshift columns are stored in blocks. You want to fit roughly the same number of column values per block across all your columns. But if you have one column where a small number of values can fit in a block, the rest of the columns end up leaving most of the block space unused. The result is wasted disk space and poor query performance.
This Postgres extension has similar-sounding storage ideas with stripes, but it’s not clear to me if it suffers from the same issue.
My first test to vet this would be a table with 50 columns of ints and one column of md5 hashes stored as varchar.
Re: Citus 10 brings columnar compression to Postgres
#6I'm curious to see how this compares in real life to TimescaleDB hypertables with compression - which to me, reads as much the same thing. I'm wondering if Citus is bringing a lower level implementation of idea possibly?
Came here to say this - I was looking to see how compression compared to timescale’s stated 91% compression. https://docs.timescale.com/latest/using-timescaledb/compress...
There are a lot of differences that need to be taken into account before making a comparison.
1. TimescaleDB implements the compression on a hifher level, the underlying table storage/access method remains the same
2. TimescaleDB doesn't compress latest data, allowing you to keep fast writes and edits for recent data, but also allows you to benefit from compression on row based data
3. Although not currently available, it is possible to have a TimescaleDB hypertable with a column based access method
4. Comparing would have to take into account the data model, access methods (types of queries), ingestion vs query comparison (batch vs real time), backfilling and editing, etc
I agree that this (Columnar) would be closer to Parquet.
Re: Citus 10 brings columnar compression to Postgres
#7One of the gotchas of columnar storage (coming from Redshift) is that you lose all of the compression benefits if you have just one column that’s fat or hard to compress. In Redshift columns are stored in blocks. You want to fit roughly the same number of column values per block across all your columns. But if you have one column where a small number of values can fit in a block, the rest of the columns end up leavin…
In citus columnar, stripes/chunks are variable in size. If you have one large column that doesn't bloat the other columns.
Re: Citus 10 brings columnar compression to Postgres
#8One of the gotchas of columnar storage (coming from Redshift) is that you lose all of the compression benefits if you have just one column that’s fat or hard to compress. In Redshift columns are stored in blocks. You want to fit roughly the same number of column values per block across all your columns. But if you have one column where a small number of values can fit in a block, the rest of the columns end up leavin…
In citus columnar, stripes/chunks are variable in size. If you have one large column that doesn't bloat the other columns.
Cool! So if I select two columns with a predicate in the first, does it scan all stripes for the second column?
Re: Citus 10 brings columnar compression to Postgres
#9Sensage/Addamark was too early to the columnar storage game in 2001-2003 ... https://en.wikipedia.org/wiki/Sensage.
Re: Citus 10 brings columnar compression to Postgres
#10One of the gotchas of columnar storage (coming from Redshift) is that you lose all of the compression benefits if you have just one column that’s fat or hard to compress. In Redshift columns are stored in blocks. You want to fit roughly the same number of column values per block across all your columns. But if you have one column where a small number of values can fit in a block, the rest of the columns end up leavin…
Do you have a source for this, or a code sample that can demonstrate it? This would be an extremely naive implementation of columnar storage. There are some truly hard cases around long variable-length strings, but any halfway decent columnar storage engine should be able to handle columns with different widths.