I love postgres and have created my own "data lake" sorta systems -- what would this add to my workflows?
Pg_lake: Postgres with Iceberg and data lake access
11–20 of 126 posts
Re: Pg_lake: Postgres with Iceberg and data lake access
#12Curious why pgduck_server is a totally separate process?
What has been pointed out from the README; also:
- Separation of concerns, since with a single external process we can share object store caches without complicated locking dances between multiple processes. - Memory limits are easier to reason about with a single external process. - Postgres backends end up being more robust, as you can restart the pgduck_server process separately.
Re: Pg_lake: Postgres with Iceberg and data lake access
#13Does anyone know how access control works to the underlying s3 objects? I didn’t see anything regarding grants in the docs.
Hi, one of the developers here. You define credentials that can access the S3 buckets and use those as DuckDB secrets, usually in an init script for pgduck_server. (You can see some examples of this in the testing framework.) I'll see if we can improve the docs or highlight that part better, if it is already documented—we did move some things around prior to release.
Maybe this could help: https://github.com/Snowflake-Labs/pg_lake?tab=readme-ov-file...
Re: Pg_lake: Postgres with Iceberg and data lake access
#14Why would Snowflake develop and release this? Doesn't this cannibalize their main product?
It's not going to scale as well as Snowflake, but it gets you into an Iceberg ecosystem which Snowflake can ingest and process at scale. Analytical data systems are typically trending to heterogenous compute with a shared storage backend -- you have large, autoscaling systems to process the raw data down to something that is usable by a smaller, cheaper query engine supporting UIs/services.
Re: Pg_lake: Postgres with Iceberg and data lake access
#15I was going to ask if you could then put DuckDB over Postgres for the OLAP query engine -- looks like that's already what it does! very interesting development in the data lake space alongside DuckLake and things
You create foreign tables in postgres using either the pg_lake_table wrapper or pg_lake_iceberg.
Once those tables exist, queries against them are able to either push down entirely to the remote tables and uses a Custom Scan to execute and pull results back into postgres, or we transform/extract the pieces that can be executed remotely using a FDW and then treat it as a tuple source.
In both cases, the user does not need to know any of the details and just runs queries inside postgres as they always have.
Re: Pg_lake: Postgres with Iceberg and data lake access
#16Man, we are living in the golden era of PostgreSQL.
Re: Pg_lake: Postgres with Iceberg and data lake access
#17Does anyone know how access control works to the underlying s3 objects? I didn’t see anything regarding grants in the docs.
Hi, one of the developers here. You define credentials that can access the S3 buckets and use those as DuckDB secrets, usually in an init script for pgduck_server. (You can see some examples of this in the testing framework.) I'll see if we can improve the docs or highlight that part better, if it is already documented—we did move some things around prior to release.
Interesting. I am working on a project to integrate access management to iceberg/parquet files for sagemaker. Controlling what users logged into sagemaker studio have access to in s3. It’s fine using static policies for mvp, but eventually it needs to be dynamic and integrated into enterprise iam tools. Those tools generally have great support for managing sql grants. Not so much for s3 bucket policies.
Re: Pg_lake: Postgres with Iceberg and data lake access
#18When Snowflake bought Crunchy Data I was hoping they were going to offer a managed version of this
It's great that I can run this locally in a Docker container, I'd love to be able to run a managed instance on AWS billed through our existing Snowflake account
Re: Pg_lake: Postgres with Iceberg and data lake access
#19Why would Snowflake develop and release this? Doesn't this cannibalize their main product?
It's not going to scale as well as Snowflake, but it gets you into an Iceberg ecosystem which Snowflake can ingest and process at scale. Analytical data systems are typically trending to heterogenous compute with a shared storage backend -- you have large, autoscaling systems to process the raw data down to something that is usable by a smaller, cheaper query engine supporting UIs/services.
But if you are used to this type of compute per dollar what on earth would make you want to move to Snowflake?
Re: Pg_lake: Postgres with Iceberg and data lake access
#20Earlier quoted context omitted.
Hi, one of the developers here. You define credentials that can access the S3 buckets and use those as DuckDB secrets, usually in an init script for pgduck_server. (You can see some examples of this in the testing framework.) I'll see if we can improve the docs or highlight that part better, if it is already documented—we did move some things around prior to release.
Interesting. I am working on a project to integrate access management to iceberg/parquet files for sagemaker. Controlling what users logged into sagemaker studio have access to in s3. It’s fine using static policies for mvp, but eventually it needs to be dynamic and integrated into enterprise iam tools. Those tools generally have great support for managing sql grants. Not so much for s3 bucket policies.
DuckDB secrets management supports custom IAM roles and the like; at this point we are basically treating the pgduck_server external system as a black box.
For the postgres grants themselves, we provide privs to allow read/write to the remote tables, which is done via granting the `pg_lake_read`, `pg_lake_write` or `pg_lake_read_write` grants. This is a blanket all-or-nothing grant, however, so would need some design work/patching to support per-relation grants, say.
(You could probably get away with making roles in postgres that have the appropriate read/write grant, then only granting those specific roles to a given relation, so it's probably doable though a little clunky at the moment.)