Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pg_lakehouse: Query Any Data Lake from Postgres

github.com

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#51

Post by sgt »

philippemnoel wrote: post #35
epsilonic wrote: post #33

How does this compare to Hydra? https://www.hydra.so/

You can see performance comparison to Hydra on ClickBench: https://benchmark.clickhouse.com/ by selecting ParadeDB and Hydra. Tl;dr: It is much faster. From a feature-set perspective, in addition to querying local disk, we can query remote object stores (S3, GCS, etc.), table format providers (Delta Lake, soon Iceberg too). From a code perspective, we're written in Rust on top of open-source standards like OpenDAL an…

And when will you have GCS storage ready? I saw on the website that it is not yet available.

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#52

Post by dewey »

whalesalad wrote: post #41

How many folks here struggle to adopt tooling like this because it isn’t possible to add psql extensions to places like RDS?

Yep, that usually dampens my excitement pretty quickly after seeing a new extension. You‘d also not know if it’s available on new versions. Sometimes you can install them „manually“ like supabase audit but more often that’s not possible.

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#53

Post by ahachete »

The (internal) use of DataFusion to create new, powerful extensions for Postgres is a very clever idea. Very good work for the ParadeDB team.

I like this one very much. Very simple way to avoid having to use different set of tools and query languages (or more limited query languages) to query lakes.

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#54

Post by ahachete »

philippemnoel wrote: post #21
nikita wrote: post #19

This is great work! Could you please comment on the choice of your license. Lost Postgres extension that achieve wide adoption use Postgres, MIT or Apache license.

All ParadeDB extensions are released under AGPL-3.0. We've found that it strikes the right balance between being open-source and enabling the community to adopt for free, while also protecting us from hyperscalers and enabling us to build a sustainable business. Perhaps the topic of a blog post someday :)

I applaud the decision to use AGPL-3.0.

For me, it's a license that provides forward guarantees to the Community: no proprietary forks can happen, so any fork will be an OSS fork from which the upstream project may benefit too, which benefits all users.

That's the reason we chose this license for StackGres [1], another project in the Postgres space.

[1]: https://stackgres.io

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#55

Post by snthpy »

nathanwallace wrote: post #18

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.) 1 - https://github.com/turbot/steampipe

I like steampipe but found the use of postgres a bit heavy for my use cases.

Could you make it run with pg_lite in wasm or DuckDB?

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#56

Post by nathanwallace »

snthpy wrote: post #55
nathanwallace wrote: post #18

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.) 1 - https://github.com/turbot/steampipe

I like steampipe but found the use of postgres a bit heavy for my use cases. Could you make it run with pg_lite in wasm or DuckDB?

In addition to Postgres FDWs, Steampipe plugins are also available as a SQLite extension [1] or a CLI export tool [2] for lighter weight use cases. (Although a local Postgres has a surprisingly small footprint!) Building plugins as DuckDB extensions would be cool too, but we haven't done that yet.

1 - https://steampipe.io/blog/2023-12-sqlite-extensions 2 - https://steampipe.io/blog/2023-12-steampipe-export

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#57

Post by xuanwo »

mustafabal wrote: post #40

Very nice addition! Do you plan to support Snowflake as an object store in the near future? It's not currently in pg_lakehouse's README.

Hi, OpenDAL's maintainer here. I'm not sure what "Snowflake as an object store" means since Snowflake is a cloud data warehouse service and not intended for storage services.

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#58

Post by thenaturalist »

pid-1 wrote: post #47
whalesalad wrote: post #41

How many folks here struggle to adopt tooling like this because it isn’t possible to add psql extensions to places like RDS?

Moreover, even when extensions are supported by RDS, they often make upgrading database versions a PITA.

Not familiar with the process, how do they make it a PITA?

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#59

Post by adammarples »

arduanika wrote: post #27
pas wrote: post #25

Earlier quoted context omitted.

Sorry, what do you base that on? To me it just seems like a straightforward inspiration from the "data lake" -> "lakehouse" terminology that Databricks started (?) using. https://www.databricks.com/product/data-lakehouse edit: ah, but in a different comment someone noted that it's not actually a lakehouse, so who knows!? :)

Based on pure speculation. I may be reaching. My best guess is that Databricks and Pg_lakehouse both independently coined "lakehouse" from "data lake", and that for the latter team, it was partly a pun on Wodehouse. But the creators are welcome to chime in and confirm/deny! (Or to say, like, "Sure...uh, we totally meant that...yes we are very literary.")

It has absolutely nothing to do with Pg wodehouse, it's a "data warehouse" sitting on a "data lake", which simply combines those two terms while keeping the "lake" and "house" metaphors quite well because you do actually get lake houses on lakes.

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#60

Post by retakeming »

nikita wrote: post #37

I have another question. So far on the clickbench leaderboard it's 15x slower than baseline. The number 1 place is 1.67 slower the baseline. I assume that's DataFusion speed. What's the plan to improve upon it?

Could you clarify which result you're referring to as the baseline and "number 1 place?"

I should clarify that our published Clickbench results are from our pg_analytics extension. New results with pg_lakehouse will be released. They're going to beat the old benchmarks because 1. No overhead from Postgres transactions/MVCC, since pg_analytics used the table access method whereas pg_lakehouse is just a foreign data wrapper 2. Uses the latest release of DataFusion.

The performance differences that exist between DataFusion and other OLAP engine are rapidly becoming commoditized. DataFusion is already a world-class query engine and will only improve. pg_lakehouse absorbs all those improvements into Postgres.

Post reply on HN