Pg_lake: Postgres with Iceberg and data lake access
1–10 of 126 posts
Re: Pg_lake: Postgres with Iceberg and data lake access
#2This is huge!
When people ask me what’s missing in the Postgres market, I used to tell them “open source Snowflake.”
Crunchy’s Postgres extension is by far the most ahead solution in the market.
Huge congrats to Snowflake and the Crunchy team on open sourcing this.
Re: Pg_lake: Postgres with Iceberg and data lake access
#3With S3 Table Buckets, Cloudflare R2 Data Catalog and now this, Iceberg seems to be winning.
Re: Pg_lake: Postgres with Iceberg and data lake access
#4I was going to ask if you could then put DuckDB over Postgres for the OLAP query engine -- looks like that's already what it does! very interesting development in the data lake space alongside DuckLake and things
Re: Pg_lake: Postgres with Iceberg and data lake access
#5Curious why pgduck_server is a totally separate process?
Re: Pg_lake: Postgres with Iceberg and data lake access
#6Curious why pgduck_server is a totally separate process?
from the README:
> This separation also avoids the threading and memory-safety limitations that would arise from embedding DuckDB directly inside the Postgres process, which is designed around process isolation rather than multi-threaded execution. Moreover, it lets us interact with the query engine directly by connecting to it using standard Postgres clients.
Re: Pg_lake: Postgres with Iceberg and data lake access
#7Curious why pgduck_server is a totally separate process?
The README explains it:
> This separation also avoids the threading and memory-safety limitations that would arise from embedding DuckDB directly inside the Postgres process, which is designed around process isolation rather than multi-threaded execution. Moreover, it lets us interact with the query engine directly by connecting to it using standard Postgres clients.
Re: Pg_lake: Postgres with Iceberg and data lake access
#8Why would Snowflake develop and release this? Doesn't this cannibalize their main product?
Re: Pg_lake: Postgres with Iceberg and data lake access
#9Does anyone know how access control works to the underlying s3 objects? I didn’t see anything regarding grants in the docs.
Re: Pg_lake: Postgres with Iceberg and data lake access
#10Does anyone know how access control works to the underlying s3 objects? I didn’t see anything regarding grants in the docs.
Hi, one of the developers here. You define credentials that can access the S3 buckets and use those as DuckDB secrets, usually in an init script for pgduck_server. (You can see some examples of this in the testing framework.)
I'll see if we can improve the docs or highlight that part better, if it is already documented—we did move some things around prior to release.