How many folks here struggle to adopt tooling like this because it isn’t possible to add psql extensions to places like RDS?
Pg_lakehouse: Query Any Data Lake from Postgres
41–50 of 73 posts
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#42Yet another amazing postgres plugin made possible by pgrx (https://github.com/pgcentralfoundation/pgrx)
It's really crazy how some projects just instantly enable a whole generation of new possibilities.
If you are impressed like this and want to build something like it -- check out pgrx, it's a pretty great experience.
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#43Earlier quoted context omitted.
There isn't a winner and there likely won't be one (at least not for a long time). Tabular will likely be acquired by Snowflake and the two industry behemoths now back their own formats, and each will treat their own as a first class citizen.
Agreed, this is why we want to support both. Maybe even Apache Hudi down the line. But I hope the industry converges to a main standard rather than Snowflake/Databricks fighting for their own formats. They can differentiate on much more meaningful features
There’s a lot of interesting work happening in this area (see: XTable).
We are building a Python distributed query engine, and share a lot of the same frustrations… in fact until quite recently most of the table formats only had JVM client libraries and so integrating it purely natively with Daft was really difficult.
We finally managed to get read integrations across Iceberg/DeltaLake/Hudi recently as all 3 now have Python/Rust-facing APIs. Funny enough, the only non-JVM implementation of Hudi was contributed by the Hudi team and currently still lives in our repo :D (https://github.com/Eventual-Inc/Daft/tree/main/daft/hudi/pyh...)
It’s still the case that these libraries still lag behind their JVM counterparts though, so it’s going to be a while before we see full support across the full featureset of each table format. But we’re definitely seeing a large appetite for working with table formats outside of the JVM ecosystem (e.g. in Python and Rust)
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#44Earlier quoted context omitted.
Agreed, this is why we want to support both. Maybe even Apache Hudi down the line. But I hope the industry converges to a main standard rather than Snowflake/Databricks fighting for their own formats. They can differentiate on much more meaningful features
There’s a lot of interesting work happening in this area (see: XTable). We are building a Python distributed query engine, and share a lot of the same frustrations… in fact until quite recently most of the table formats only had JVM client libraries and so integrating it purely natively with Daft was really difficult. We finally managed to get read integrations across Iceberg/DeltaLake/Hudi recently as all 3 now have…
Are you using the iceberg-rust crate for Rust? It's a rather young project, have you found it sufficient for your needs (if using)?
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#45Yet another amazing postgres plugin made possible by pgrx ( https://github.com/pgcentralfoundation/pgrx ) It's really crazy how some projects just instantly enable a whole generation of new possibilities. If you are impressed like this and want to build something like it -- check out pgrx, it's a pretty great experience.
pgrx is indeed wonderful and we would not be able to do our work without it. Big kudos to Eric, Jubilee and rest of team!
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#46Very nice addition! Do you plan to support Snowflake as an object store in the near future? It's not currently in pg_lakehouse's README.
Snowflake is not in the list of supported stores on Apache OpenDAL, so likely not. It might not expose its storage APIs. I doubt users of Snowflake would want a separate query engine anyways
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#47How many folks here struggle to adopt tooling like this because it isn’t possible to add psql extensions to places like RDS?
Moreover, even when extensions are supported by RDS, they often make upgrading database versions a PITA.
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#48How does this compare to Hydra? https://www.hydra.so/
You can see performance comparison to Hydra on ClickBench: https://benchmark.clickhouse.com/ by selecting ParadeDB and Hydra. Tl;dr: It is much faster. From a feature-set perspective, in addition to querying local disk, we can query remote object stores (S3, GCS, etc.), table format providers (Delta Lake, soon Iceberg too). From a code perspective, we're written in Rust on top of open-source standards like OpenDAL an…
Thanks for the prompt response, the support for OpenDAL is amazing!
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#49Earlier quoted context omitted.
Whereas pg_analytics stores the data in Postgres block storage, pg_lakehouse does not use Postgres storage at all. This makes it a much simpler (and in our opinion, more elegant) extension. We learned that many of our users already stored their Parquet files in S3, so it made sense to connect directly to S3 rather than asking them to ingest those Parquet files into Postgres. It also accelerates the path to production…
If users are already having datalake kind of system which is generating parquet files, the use case to use Postgres to query the data itself is questionable. I think having Postgres way of doing things should be prioritised if you want to keep your product in unique position.
Can you elaborate on what you mean by the "Postgres way of doing things"? Also, what is wrong with using Postgres to query data in external object stores? It is a common occurrence for businesses to store parquet artefacts in object storage, and querying them is often desirable.
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#50I am not up to date in various lakes. Is this read-only? Are you able to init a lake from scratch?
What's the model to feed such a lake from some queue?