Nice does this also allow me to write to parquet from my Postgres table?
Pg_lake: Postgres with Iceberg and data lake access
61–70 of 126 posts
Re: Pg_lake: Postgres with Iceberg and data lake access
#62Man, we are living in the golden era of PostgreSQL.
[deleted]
Re: Pg_lake: Postgres with Iceberg and data lake access
#63Can someone dumb this down a bit for a non data-engineer? Hard to fully wrap my head around who this is/isn’t best suited for.
Re: Pg_lake: Postgres with Iceberg and data lake access
#64Nice does this also allow me to write to parquet from my Postgres table?
Ong yes it works this would have made my past job so much easier.
Re: Pg_lake: Postgres with Iceberg and data lake access
#65Nice does this also allow me to write to parquet from my Postgres table?
Yes, just COPY table TO 's3://mybucket/data.parquet'
Or COPY table TO STDOUT WITH (format 'parquet') if you need it on the client side.
Re: Pg_lake: Postgres with Iceberg and data lake access
#66Why would Snowflake develop and release this? Doesn't this cannibalize their main product?
It'll probably be really difficult to set up. If it's anything like super base, your question the existence of God when trying to get it to work properly. You pay them to make it work right.
For testing, we at least have a Dockerfile to automate the setup of the pgduck_server and a minio instance so it Just Works™ with the extensions installed in your local Postgres cluster (after installing the extensions).
The configuration mainly involves just defining the default iceberg location for new tables, pointing it to the pgduck_server, and providing the appropriate auth/secrets for your bucket access.
Re: Pg_lake: Postgres with Iceberg and data lake access
#67Why would Snowflake develop and release this? Doesn't this cannibalize their main product?
When we first developed pg_lake at Crunchy Data and defined GTM we considered whether it could be a Snowflake competitor, but we quickly realised that did not make sense.
Data platforms like Snowflake are built as a central place to collect your organisation's data, do governance, large scale analytics, AI model training and inference, share data within and across orgs, build and deploy data products, etc. These are not jobs for a Postgres server.
Pg_lake foremost targets Postgres users who currently need complex ETL pipelines to get data in and out of Postgres, and accidental Postgres data warehouses where you ended up overloading your server with slow analytical queries, but you still want to keep using Postgres.
Re: Pg_lake: Postgres with Iceberg and data lake access
#68Can someone dumb this down a bit for a non data-engineer? Hard to fully wrap my head around who this is/isn’t best suited for.
One usecase we have (we built it ourselves) is to periodically offload data from Postgres to lake house partitioned data on GCS. The way I see it this can now be done with a single query. Another one is the other way around to use posters as a query engine or to merge offloaded data with your live data.
Re: Pg_lake: Postgres with Iceberg and data lake access
#69Crunchydata did it first :) but nice to get more options
Re: Pg_lake: Postgres with Iceberg and data lake access
#70Crunchydata did it first :) but nice to get more options
It's the same team and same project :). Crunchy Data was acquired by Snowflake.