Interesting! How does it compare with ducklake?
Pg_lake: Postgres with Iceberg and data lake access
21–30 of 126 posts
Re: Pg_lake: Postgres with Iceberg and data lake access
#22Interesting! How does it compare with ducklake?
You could say
With DuckLake, the query frontend and query engine are DuckDB, and Postgres is used as a catalog in the background.
With pg_lake, the query frontend and catalog are Postgres, and DuckDB is used as a query engine in the background.
Of course, they also use different table formats (though similar in data layer) with different pros and cons, and the query frontends differ in significant ways.
An interesting thing about pg_lake is that it is effectively standalone, no external catalog required. You can point Spark et al. directly to Postgres with pg_lake by using the Iceberg JDBC driver.
Re: Pg_lake: Postgres with Iceberg and data lake access
#23This is really nice though looking at the code - a lot of the postgres types are missing as well a lot of the newer parquet logical types - but this is a great start and a nice use of FDW.
Re: Pg_lake: Postgres with Iceberg and data lake access
#24This is really nice though looking at the code - a lot of the postgres types are missing as well a lot of the newer parquet logical types - but this is a great start and a nice use of FDW.
Also, any planned support for more catalogs?
Re: Pg_lake: Postgres with Iceberg and data lake access
#25Does anyone know how access control works to the underlying s3 objects? I didn’t see anything regarding grants in the docs.
There are Postgres roles for read/write access to the S3 object that DuckDB has access to. Those roles can create tables from specific files or at specific locations, and can then assign more fine-grained privileges to other Postgres roles (e.g. read access on a specific view or table).
Re: Pg_lake: Postgres with Iceberg and data lake access
#26Curious why pgduck_server is a totally separate process?
from the README: > This separation also avoids the threading and memory-safety limitations that would arise from embedding DuckDB directly inside the Postgres process, which is designed around process isolation rather than multi-threaded execution. Moreover, it lets us interact with the query engine directly by connecting to it using standard Postgres clients.
Thanks! Didn’t scroll down far enough
Re: Pg_lake: Postgres with Iceberg and data lake access
#27Why would Snowflake develop and release this? Doesn't this cannibalize their main product?
One thing I admire about Snowflake is a real commitment to self-cannibalization. They were super out front with Iceberg even though it could disrupt them, because that's what customers were asking for and they're willing to bet they'll figure out how to make money in that new world
Video of their SVP of Product talking about it here: https://youtu.be/PERZMGLhnF8?si=DjS_OgbNeDpvLA04&t=1195
Re: Pg_lake: Postgres with Iceberg and data lake access
#28This is really nice though looking at the code - a lot of the postgres types are missing as well a lot of the newer parquet logical types - but this is a great start and a nice use of FDW.
Hi, what types are you expecting to see that aren't supported? I believe we had support for most/all builtin postgres types.
Re: Pg_lake: Postgres with Iceberg and data lake access
#29I was going to ask if you could then put DuckDB over Postgres for the OLAP query engine -- looks like that's already what it does! very interesting development in the data lake space alongside DuckLake and things
You create foreign tables in postgres using either the pg_lake_table wrapper or pg_lake_iceberg. Once those tables exist, queries against them are able to either push down entirely to the remote tables and uses a Custom Scan to execute and pull results back into postgres, or we transform/extract the pieces that can be executed remotely using a FDW and then treat it as a tuple source. In both cases, the user does not…
I think I don't understand postgres enough, so forgive this naive question, but what does pushing down to the remote tables mean? Does it allow parallelism? If I query a very large iceberg table, will this system fan the work out to multiple duckdb executors and gather the results back in?
Re: Pg_lake: Postgres with Iceberg and data lake access
#30Earlier quoted context omitted.
It's not going to scale as well as Snowflake, but it gets you into an Iceberg ecosystem which Snowflake can ingest and process at scale. Analytical data systems are typically trending to heterogenous compute with a shared storage backend -- you have large, autoscaling systems to process the raw data down to something that is usable by a smaller, cheaper query engine supporting UIs/services.
But if you are used to this type of compute per dollar what on earth would make you want to move to Snowflake?
Different parts of the analytical stack have different performance requirements and characteristics. Maybe none of your stack needs it and so you never need Snowflake at all.
More likely, you don't need Snowflake to process queries from your BI tools (Mode, Tableau, Superset, etc), but you do need it to prepare data for those BI tools. Its entirely possible that you have hundreds of terabytes, if not petabytes, of input data that you want to pare down to You shouldn't really consider analytical systems the same as a database backing a service. Analytical systems are designed to funnel large datasets that cover the entire business (cross cutting services and any sharding you've done) into subsequently smaller datasets that are cheaper and faster to query. And you may be using different compute engines for different parts of these pipelines; there's a good chance you're not using only Snowflake but Snowflake and a bunch of different tools.