Looks like pg as a replacement for databricks sql, which is already a query engine for datalakes. It's not a lakehouse, but it calls itself one. Seems like a cool and useful project, but the name is problematic.
Pg_lakehouse: Query Any Data Lake from Postgres
11–20 of 73 posts
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#12It seems very promising!
2 questions:
- do you distribute query processing over multiple pg nodes ?
- do you store the metadata in PG, instead of a traditional metastore?
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#13Very cool! Could you share the key difference between this and the previous pg_analytics, and motivation of making it a separate plugin?
Whereas pg_analytics stores the data in Postgres block storage, pg_lakehouse does not use Postgres storage at all.
This makes it a much simpler (and in our opinion, more elegant) extension. We learned that many of our users already stored their Parquet files in S3, so it made sense to connect directly to S3 rather than asking them to ingest those Parquet files into Postgres.
It also accelerates the path to production readiness, since we're not touching Postgres internals (no need to mess with Postgres MVCC, write ahead logs, transactions, etc.)
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#14It seems very promising! 2 questions: - do you distribute query processing over multiple pg nodes ? - do you store the metadata in PG, instead of a traditional metastore?
Thanks!
1. It's single node, but DataFusion parallelizes query execution across multiple cores. We do have plans for a distributed architecture, but we've found that you can get ~very~ far just by scaling up a single Postgres node.
2. The only information stored in Postgres are the options passed into the foreign data wrapper and the schema of the foreign table (this is standard for all Postgres foreign data wrappers).
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#15Neat that you plan to support both Delta Lake and Apache Iceberg I'm curious about HN's position between these two formats? I'm having a hard time deciphering which might be the industry winner (or perhaps they both have a place, no "winner" necessary)
I'm also building in the lakehouse space and anecdotally have seen more excitement around Iceberg over delta lake just because of its completely open source origins. Iceberg has evolved faster and has had more contributions from a more diverse set of contributors than Delta Lake. Not sure if this will change with a Snowflake Tabular acquisition but I'd easily bet on Iceberg if current trends continue.
We agree. We plan to bring Iceberg support as a first-class citizen as soon as we can, but unfortunately the support in Rust these days is still limited. We and the community are working on it
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#16Neat that you plan to support both Delta Lake and Apache Iceberg I'm curious about HN's position between these two formats? I'm having a hard time deciphering which might be the industry winner (or perhaps they both have a place, no "winner" necessary)
This is anecdotal, but I feel that we (ParadeDB) have received more requests for Iceberg integration vs. Delta Lake. We were actually hesitant to launch pg_lakehouse without Iceberg support, but pulled the trigger on it because the iceberg-rust crate is still in its early days. We will probably be contributing to iceberg-rust to make it work with pg_lakehouse.
Also anecdotal, but we (Spice AI) see more requests for Iceberg, but in practice more deployments of Delta Lake.
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#17Looks like pg as a replacement for databricks sql, which is already a query engine for datalakes. It's not a lakehouse, but it calls itself one. Seems like a cool and useful project, but the name is problematic.
pg_house just wasn't as catchy!
In all seriousness though, I see your point. While it's true that we don't provide the storage or table format, our belief is that companies actually want to own the data in their S3. We called it pg_lakehouse because it's the missing glue for companies already using Postgres + S3 + Delta Lake/Iceberg to have a lakehouse without new infrastructure.
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#18Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.)
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#19This is great work! Could you please comment on the choice of your license. Lost Postgres extension that achieve wide adoption use Postgres, MIT or Apache license.
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#20Nice. I wish timescaledb open-sourced their s3 storage thing.
They've been moving more and more towards closed source over the years, which is a shame but I understand why. We don't offer time-series features today, but we're not ruling out adding support for it eventually if it is desired by our users.