Is this deltalake the same as Databricks deltalake? Is it compatible?
Transforming Postgres into a Fast OLAP Database
11–20 of 62 posts
Re: Transforming Postgres into a Fast OLAP Database
#12Interesting and smart choice to use Apache DataFusion as their query engine.
which is point of worry, because that one is not mature and well adapted product
Re: Transforming Postgres into a Fast OLAP Database
#13If you're looking for pg_analytics (the product in TFA) in the charts and can't find it, it's because the chart calls it ParadeDB (after the company, instead of the product).
One of the ParadeDB makers here -- Yes, we release our Helm charts under the `paradedb` name.
We also ship our individual extensions (pg_analytics and pg_bm25) pre-compiled as .deb under our GitHub Releases, and as a fully complete product, ParadeDB, as a Dockerfile and Helm chart
Re: Transforming Postgres into a Fast OLAP Database
#14Interesting and smart choice to use Apache DataFusion as their query engine.
ParadeDB maker here -- DataFusion is amazing! Apple used it to rebuild part of Spark and open-sourced it under Comet too: https://github.com/apache/arrow-datafusion-comet/pull/1
It's a lot more mature than people think :)
Re: Transforming Postgres into a Fast OLAP Database
#15There are many Postgres forks for OLAP workloads, such as: - Greenplum; - Citus with cstore_fdw; - IMCS (Konstantin Knizhnik); - Hydra; - AlloyDB; For example, Greenplum is one of the earliest, fairly mature, but abandoned.
That's right! Greenplum is still active, but falling off.
We see pg_analytics as the next-generation Citus columnar, with much better performance and integration into the wider data ecosystem via Delta Lake, and eventually Iceberg
Re: Transforming Postgres into a Fast OLAP Database
#16Thanks for sharing, I really like Postgres. However, I generally use Postgres for OLTP work. I would like to point out two things: 1. Based on Clickbench results, pg_analytics is still far from top-tier performance. If you're looking for a high-performance OLAP database, you should consider the top-ranked products. 2. The queries tested by Clickbench are really simple, far from real data analysis scenarios. You must…
One of the authors of pg_analytics here.
1. On Clickbench, make sure you're doing an apples-to-apples comparison by comparing scores from the same instance. We used the most commonly-used c6a.4xlarge instance. While a few databases like DuckDB rank higher, the performance of Datafusion (our underlying query engine) is constantly improving, and pg_analytics inherits those improvements.
Then again, people only care about performance and benchmarks up to a certain threshold. The goal of pg_analytics is not to displace something like StarRocks, but to enable analytical workloads that require both row and column-oriented data or Postgres transactions.
2. We're working on TPC-H benchmarks. They're good for demonstrating JOIN performance and we'll have them published early next week.
Re: Transforming Postgres into a Fast OLAP Database
#17clickhouse is phenomenal
Re: Transforming Postgres into a Fast OLAP Database
#18Is this deltalake the same as Databricks deltalake? Is it compatible?
Yes. delta-rs is a Rust-based implementation of Delta Lake.
The existing version of pg_analytics uses delta-rs to manage Parquet files stored within Postgres. In the future, we plan on integrating external object stores. This means that you'll be able to query any Delta Lake directly from Postgres.
Iceberg support will come later, once the Rust implementation of Iceberg matures.
Re: Transforming Postgres into a Fast OLAP Database
#19Thanks for sharing, I really like Postgres. However, I generally use Postgres for OLTP work. I would like to point out two things: 1. Based on Clickbench results, pg_analytics is still far from top-tier performance. If you're looking for a high-performance OLAP database, you should consider the top-ranked products. 2. The queries tested by Clickbench are really simple, far from real data analysis scenarios. You must…
here's a link comparing parade to duckdb, clickhouse and snowflake
https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQi...
Re: Transforming Postgres into a Fast OLAP Database
#20If I combined ParadeDB with a common postgres extension like PostGIS, would I get the benefits of both?