Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pg_lakehouse: Query Any Data Lake from Postgres

github.com

Post reply on HN
Print view View original

31–40 of 73 posts

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#31

Post by francoismassot »

philippemnoel wrote: post #21
nikita wrote: post #19

This is great work! Could you please comment on the choice of your license. Lost Postgres extension that achieve wide adoption use Postgres, MIT or Apache license.

All ParadeDB extensions are released under AGPL-3.0. We've found that it strikes the right balance between being open-source and enabling the community to adopt for free, while also protecting us from hyperscalers and enabling us to build a sustainable business. Perhaps the topic of a blog post someday :)

Well, MongoDB was under AGPL v3.0 :)

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#32

Post by FridgeSeal »

lukekim wrote: post #16
retakeming wrote: post #6

Earlier quoted context omitted.

This is anecdotal, but I feel that we (ParadeDB) have received more requests for Iceberg integration vs. Delta Lake. We were actually hesitant to launch pg_lakehouse without Iceberg support, but pulled the trigger on it because the iceberg-rust crate is still in its early days. We will probably be contributing to iceberg-rust to make it work with pg_lakehouse.

Also anecdotal, but we (Spice AI) see more requests for Iceberg, but in practice more deployments of Delta Lake.

My theory is that everyone would _prefer_ to use Iceberg, but isn’t as widely supported _yet_, so they’re stuck with Delta in the interim.

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#35

Post by philippemnoel »

epsilonic wrote: post #33

How does this compare to Hydra? https://www.hydra.so/

You can see performance comparison to Hydra on ClickBench: https://benchmark.clickhouse.com/ by selecting ParadeDB and Hydra. Tl;dr: It is much faster.

From a feature-set perspective, in addition to querying local disk, we can query remote object stores (S3, GCS, etc.), table format providers (Delta Lake, soon Iceberg too).

From a code perspective, we're written in Rust on top of open-source standards like OpenDAL and DataFusion, while Hydra is their own codebase built from a fork of Citus columnar, in C.

Hydra is a cool project. Hope this helps! :)

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#36

Post by fulmicoton »

nikita wrote: post #24
philippemnoel wrote: post #21

Earlier quoted context omitted.

All ParadeDB extensions are released under AGPL-3.0. We've found that it strikes the right balance between being open-source and enabling the community to adopt for free, while also protecting us from hyperscalers and enabling us to build a sustainable business. Perhaps the topic of a blog post someday :)

It looks like hyper scalers can still host it as long as they are publishing changes to the source code ? Am I reading the license right ?

Yes

Re: Pg_lakehouse: Query Any Data Lake from Postgres

#38

Post by nitinreddy88 »

retakeming wrote: post #13
sdairs wrote: post #8

Very cool! Could you share the key difference between this and the previous pg_analytics, and motivation of making it a separate plugin?

Whereas pg_analytics stores the data in Postgres block storage, pg_lakehouse does not use Postgres storage at all. This makes it a much simpler (and in our opinion, more elegant) extension. We learned that many of our users already stored their Parquet files in S3, so it made sense to connect directly to S3 rather than asking them to ingest those Parquet files into Postgres. It also accelerates the path to production…

If users are already having datalake kind of system which is generating parquet files, the use case to use Postgres to query the data itself is questionable. I think having Postgres way of doing things should be prioritised if you want to keep your product in unique position.
Post reply on HN