How many folks here struggle to adopt tooling like this because it isn’t possible to add psql extensions to places like RDS?
We're working on getting our extensions approved on as many platforms as possible
Pg_lakehouse: Query Any Data Lake from Postgres
61–70 of 73 posts
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#62Earlier quoted context omitted.
Also anecdotal, but we (Spice AI) see more requests for Iceberg, but in practice more deployments of Delta Lake.
My theory is that everyone would _prefer_ to use Iceberg, but isn’t as widely supported _yet_, so they’re stuck with Delta in the interim.
I think you're spot on
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#63I am not up to date in various lakes. Is this read-only? Are you able to init a lake from scratch? What's the model to feed such a lake from some queue?
For now it is read-only, but soon will be write-supported too. You can feed data via Kafka
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#64Earlier quoted context omitted.
You can see performance comparison to Hydra on ClickBench: https://benchmark.clickhouse.com/ by selecting ParadeDB and Hydra. Tl;dr: It is much faster. From a feature-set perspective, in addition to querying local disk, we can query remote object stores (S3, GCS, etc.), table format providers (Delta Lake, soon Iceberg too). From a code perspective, we're written in Rust on top of open-source standards like OpenDAL an…
And when will you have GCS storage ready? I saw on the website that it is not yet available.
Next week (May 20-something)
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#65Earlier quoted context omitted.
All ParadeDB extensions are released under AGPL-3.0. We've found that it strikes the right balance between being open-source and enabling the community to adopt for free, while also protecting us from hyperscalers and enabling us to build a sustainable business. Perhaps the topic of a blog post someday :)
I applaud the decision to use AGPL-3.0. For me, it's a license that provides forward guarantees to the Community: no proprietary forks can happen, so any fork will be an OSS fork from which the upstream project may benefit too, which benefits all users. That's the reason we chose this license for StackGres [1], another project in the Postgres space. [1]: https://stackgres.io
Another great benefit of it indeed
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#66Earlier quoted context omitted.
Whereas pg_analytics stores the data in Postgres block storage, pg_lakehouse does not use Postgres storage at all. This makes it a much simpler (and in our opinion, more elegant) extension. We learned that many of our users already stored their Parquet files in S3, so it made sense to connect directly to S3 rather than asking them to ingest those Parquet files into Postgres. It also accelerates the path to production…
If users are already having datalake kind of system which is generating parquet files, the use case to use Postgres to query the data itself is questionable. I think having Postgres way of doing things should be prioritised if you want to keep your product in unique position.
It depends. If you're happy with Databricks, etc. you might be good. But we've seen many users want the simplicity of querying data from Postgres for analytics, especially in case of JOINing both analytics and transactional data
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#67Earlier quoted context omitted.
And when will you have GCS storage ready? I saw on the website that it is not yet available.
Next week (May 20-something)
Very exciting. We use GCS but we don't have a data lake yet. If you were to "sell" the concept of a data lake, what would you refer to as great example usecase?
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#68I have another question. So far on the clickbench leaderboard it's 15x slower than baseline. The number 1 place is 1.67 slower the baseline. I assume that's DataFusion speed. What's the plan to improve upon it?
Could you clarify which result you're referring to as the baseline and "number 1 place?" I should clarify that our published Clickbench results are from our pg_analytics extension. New results with pg_lakehouse will be released. They're going to beat the old benchmarks because 1. No overhead from Postgres transactions/MVCC, since pg_analytics used the table access method whereas pg_lakehouse is just a foreign data wr…
Would be great to also see new pg_lakehouse and datafusion benchmark results here: https://duckdblabs.github.io/db-benchmark/
Currently Datafusion is much slower than duckdb or OOMing.
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#69Earlier quoted context omitted.
There’s a lot of interesting work happening in this area (see: XTable). We are building a Python distributed query engine, and share a lot of the same frustrations… in fact until quite recently most of the table formats only had JVM client libraries and so integrating it purely natively with Daft was really difficult. We finally managed to get read integrations across Iceberg/DeltaLake/Hudi recently as all 3 now have…
Are you using the iceberg-rust crate for Rust? It's a rather young project, have you found it sufficient for your needs (if using)?
We're actually using pyiceberg to retrieve metadata! All our IO and decoding happens in the rust side once the data has been passthrough.
We expose something called a ScanOperator which allows integration into various catalogs through a thin layer that exposes ScanTasks.
Iceberg's impl: https://github.com/Eventual-Inc/Daft/blob/416009138359a9d410...
Re: Pg_lakehouse: Query Any Data Lake from Postgres
#70Earlier quoted context omitted.
I like steampipe but found the use of postgres a bit heavy for my use cases. Could you make it run with pg_lite in wasm or DuckDB?
In addition to Postgres FDWs, Steampipe plugins are also available as a SQLite extension [1] or a CLI export tool [2] for lighter weight use cases. (Although a local Postgres has a surprisingly small footprint!) Building plugins as DuckDB extensions would be cool too, but we haven't done that yet. 1 - https://steampipe.io/blog/2023-12-sqlite-extensions 2 - https://steampipe.io/blog/2023-12-steampipe-export
This is awesome!
I last played with it over a year ago and I see that blog post is from Dec 2023. Looking forward to trying out the sqlite extension.