21–30 of 62 posts
So are they using Parquet, ORC or Arrow under the hood; or do they have a custom format?
i recently started diving into postgresql and it seems to me that there is a patchwork of HA solutions (with patroni being the most feature-full) but no real multi-master solution for postgresql released under an open source license.
There's BDR (bi-directional replication) but apparently 2ndquadrant pulled it back under a proprietary license, am i right?
what's the current status of postgresql multi-master HA?
Sensage/Addamark was too early to the columnar storage game in 2001-2003 ... https://en.wikipedia.org/wiki/Sensage .
I think the claim here is that Citus added it to PostgreSQL. Depending on how one defines that claim, Greenplum may have been first. Disclosure: I work for VMware, which sponsors Greenplum development.
What we've done is add a Columnar storage feature into the Citus open source extension to Postgres, as part of Citus 10.
One way to think of it: Citus 10 now gives Postgres users (those who are also using the Citus extension) a columnar compression option, for use on a single node and/or a distributed Citus cluster.
since we're talking postgresql: i recently started diving into postgresql and it seems to me that there is a patchwork of HA solutions (with patroni being the most feature-full) but no real multi-master solution for postgresql released under an open source license. There's BDR (bi-directional replication) but apparently 2ndquadrant pulled it back under a proprietary license, am i right? what's the current status of p…
Sensage/Addamark was too early to the columnar storage game in 2001-2003 ... https://en.wikipedia.org/wiki/Sensage .
I think the claim here is that Citus added it to PostgreSQL. Depending on how one defines that claim, Greenplum may have been first. Disclosure: I work for VMware, which sponsors Greenplum development.
One of the gotchas of columnar storage (coming from Redshift) is that you lose all of the compression benefits if you have just one column that’s fat or hard to compress. In Redshift columns are stored in blocks. You want to fit roughly the same number of column values per block across all your columns. But if you have one column where a small number of values can fit in a block, the rest of the columns end up leavin…
Do you have a source for this, or a code sample that can demonstrate it? This would be an extremely naive implementation of columnar storage. There are some truly hard cases around long variable-length strings, but any halfway decent columnar storage engine should be able to handle columns with different widths.
The compression actually works as advertised, but that document outlines why you don't want to have the sortkey skew from other columns. The disk I/O balloons as it goes and fetches a larger number of blocks from the other columns that match what you're filtering in the sortkey.
Beware that simply adding a column-oriented storage engine to a row store like Postgres is not going to get you anywhere near the performance of a ground-up columnar system like Redshift or Snowflake. This paper explains why [1]. Short version: most of the benefits are in the column-oriented execution engine, which differs in every aspect of its implementation from a row-oriented execution engine. [1] https://stratos…
The design and implementation of modern column-oriented database systems - https://news.ycombinator.com/item?id=18076547 - Sept 2018 (42 comments)
Design and Implementation of Modern Column-Oriented Databases (2012) [pdf] - https://news.ycombinator.com/item?id=11803299 - May 2016 (9 comments)
Great start! Keep in mind the limitations: What are the Limitations? These limitations are not set in stone, and we look forward to working on them in the future: No UPDATE or DELETE support No index support No logical replication or logical decoding support See more limitations in the columnar README
But they could solve it with just a little bit of work: * Create a hidden bitmap information to store whether the „row“ of the columnar table is still valid * When updating/deleting values only set the bitmap information to zero to indicate the value is no longer valid * Every updated „row“ is added into a special chunk at the end of the table in uncompressed format * When vacuuming the table the old rows with bitmap=0 are deleted and the values from the special chunk are merged
So you would have same performance if update/delete is never done as the complete bitmap index is filled with 1. And every update/delete will make it just a little bit slower, as most often only old date is stored in compressed format not much uncomprossed values will be stored. And a vacuum full would optimize these old tables again.