Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Citus 10 brings columnar compression to Postgres

citusdata.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Citus 10 brings columnar compression to Postgres

#41

Post by znpy »

merb wrote: post #34
znpy wrote: post #22

since we're talking postgresql: i recently started diving into postgresql and it seems to me that there is a patchwork of HA solutions (with patroni being the most feature-full) but no real multi-master solution for postgresql released under an open source license. There's BDR (bi-directional replication) but apparently 2ndquadrant pulled it back under a proprietary license, am i right? what's the current status of p…

> i recently started diving into postgresql and it seems to me that there is a patchwork of HA solutions (with patroni being the most feature-full) but no real multi-master solution for postgresql released under an open source license. true multi master is barely needed. but there is citus which uses the gpl, which can be run in multi master. and yes patroni is really really awesome.

Thanks, I'll look into that!

Re: Citus 10 brings columnar compression to Postgres

#43

Post by glogla »

mslot wrote: post #20
georgewfraser wrote: post #11

Beware that simply adding a column-oriented storage engine to a row store like Postgres is not going to get you anywhere near the performance of a ground-up columnar system like Redshift or Snowflake. This paper explains why [1]. Short version: most of the benefits are in the column-oriented execution engine, which differs in every aspect of its implementation from a row-oriented execution engine. [1] https://stratos…

Correct, though it depends whether you are CPU-bound or I/O-bound. We see the latter a lot more often for large data sets. Columnar storage for PostgreSQL is especially relevant in cloud environments. Most database servers in the cloud use managed, network-attached disks because of durability, availability, and encryption-at-rest requirements. However, those do come with a performance penalty compared to local SSDs.…

That's exactly what we've seen. I don't know how Citus exactly stores data, but we found difference of 30x between gzipped Parquet and "generic Oracle table".

There's a huge difference whether your analytical query is scanning full 30 GB or 1 GB (or maybe even half that or less if you need to scan just some columns).

Re: Citus 10 brings columnar compression to Postgres

#44

Post by mrits »

teej wrote: post #5

One of the gotchas of columnar storage (coming from Redshift) is that you lose all of the compression benefits if you have just one column that’s fat or hard to compress. In Redshift columns are stored in blocks. You want to fit roughly the same number of column values per block across all your columns. But if you have one column where a small number of values can fit in a block, the rest of the columns end up leavin…

Several different compression algorithms would do very well for this. E.g dict compression for low cardinality wide columns. I'd recommend familiarizing yourself with a handful of different encodings for columnar data stores before you give up on the idea.

Re: Citus 10 brings columnar compression to Postgres

#45

Post by ablekh »

simonw wrote: post #27

Reassuring to see big new features like this coming out after the Microsoft acquisition, a healthy sign that this open source product continues to see serious ongoing investment.

Technically, Citus product is open core, not open source. Though open sourcing shard rebalancer is a recent welcome step.

Re: Citus 10 brings columnar compression to Postgres

#47

Post by GordonS »

georgewfraser wrote: post #11

Beware that simply adding a column-oriented storage engine to a row store like Postgres is not going to get you anywhere near the performance of a ground-up columnar system like Redshift or Snowflake. This paper explains why [1]. Short version: most of the benefits are in the column-oriented execution engine, which differs in every aspect of its implementation from a row-oriented execution engine. [1] https://stratos…

I agree,l that it's never going to be as effective as something built from the ground up as a column store. But the performance and storage improvements can still be immense over standard Postgres. And if you're already using Postgres, it means one less service to configure and maintain, and means you can easily combine both standard relational data with columnar data. For a lot of folks, that a huge win.

Re: Citus 10 brings columnar compression to Postgres

#48

Post by mcrutcher »

jacques_chester wrote: post #36
asah wrote: post #26

Earlier quoted context omitted.

Greenplum was founded in 2003, A/S was 2001.

Again, my qualifier is "to PostgreSQL". I couldn't see such a connection on a brief skim of the A/S wikipedia entry.

I was the PM for the software side of Greenplum DB for a while. I also don't know of an earlier example of a separate column oriented storage option on top of a Postgres base. Vertica and ParAccel (which became Redshift) both also started from a Postgres base (though Vertica claims otherwise in marketing material) but were a little later. In any case, "Column oriented storage for Postgres" is almost 20 years old. This also isn't the first open source implementation.

Re: Citus 10 brings columnar compression to Postgres

#49

Post by manigandham »

pm90 wrote: post #33
merb wrote: post #32

Earlier quoted context omitted.

well because they are the only cloud vendor with a HA master-master postgresql database (citusdb managed) it probably is also a selling point.

Wait... they have 2 read/write masters? And you can read/write from either master at the same time?

No, it creates a standby (of every node) when HA is enabled.

https://docs.microsoft.com/en-us/azure/postgresql/howto-hype...

Re: Citus 10 brings columnar compression to Postgres

#50

Post by manigandham »

merb wrote: post #32
simonw wrote: post #27

Reassuring to see big new features like this coming out after the Microsoft acquisition, a healthy sign that this open source product continues to see serious ongoing investment.

well because they are the only cloud vendor with a HA master-master postgresql database (citusdb managed) it probably is also a selling point.

Where are you seeing this multi-master feature? As far as I can tell, it uses standby nodes, not multiple active coordinators.
Post reply on HN