Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pg_lake: Postgres with Iceberg and data lake access

github.com

Post reply on HN
Print view View original

71–80 of 126 posts

Re: Pg_lake: Postgres with Iceberg and data lake access

#72

Post by barrrrald »

qaq wrote: post #49
barrrrald wrote: post #27

Earlier quoted context omitted.

One thing I admire about Snowflake is a real commitment to self-cannibalization. They were super out front with Iceberg even though it could disrupt them, because that's what customers were asking for and they're willing to bet they'll figure out how to make money in that new world Video of their SVP of Product talking about it here: https://youtu.be/PERZMGLhnF8?si=DjS_OgbNeDpvLA04&t=1195

Have you interacted with Snowflake teams much? We are using external iceberg tables with snowflake. Every interaction pretty much boils down to you really should not be using iceberg you should be using snowflake for storage. It's also pretty obvious some things are strategically not implemented to push you very strongly in that direction.

Not surprised - this stuff isn’t fully mature yet. But I interact with their team a lot and know they have a commitment to it (I’m the other guy in that video)

Re: Pg_lake: Postgres with Iceberg and data lake access

#73

Post by pgguru »

inglor wrote: post #24
inglor wrote: post #23

This is really nice though looking at the code - a lot of the postgres types are missing as well a lot of the newer parquet logical types - but this is a great start and a nice use of FDW.

Also, any planned support for more catalogs?

I think we have recently merged (or are getting ready to merge) REST catalog support, so that will open some things up in this department.

Re: Pg_lake: Postgres with Iceberg and data lake access

#76

Post by derefr »

mritchie712 wrote: post #59
mslot wrote: post #45

Earlier quoted context omitted.

DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus. DuckLake can do things that pg_lake cannot do with Iceberg, and DuckDB can do things Postgres absolutely can't (e.g. query data frames). On the other hand, Postgres can do a lot of things that DuckDB cannot do. For instance, it can handl…

> For instance, it can handle >100k single row inserts/sec. DuckLake already has data-inlining for the DuckDB catalog, seems this will be possible once it's supported in the pg catalog. > Postgres also has a more natural persistence & continuous processing story, so you can set up pg_cron jobs and use PL/pgSQL (with heap tables for bookkeeping) to do orchestration. This is true, but it's not clear where I'd use this…

> This is true, but it's not clear where I'd use this in practice. e.g. if I need to run a complex ETL job, I probably wouldn't do it in pg_cron.

Think "tiered storage."

See the example under https://github.com/Snowflake-Labs/pg_lake/blob/main/docs/ice...:

   select cron.schedule('flush-queue', '* * * * *', $$
     with new_rows as (
       delete from measurements_staging returning *
     )
     insert into measurements select * from new_rows;
   $$);
The "continuous ETL" process the GP is talking about would be exactly this kind of thing, and just as trivial. (In fact it would be this exact same code, just with your mental model flipped around from "promoting data from a staging table into a canonical iceberg table" to "evicting data from a canonical table into a historical-archive table".)

Re: Pg_lake: Postgres with Iceberg and data lake access

#77

Post by dunefox »

anktor wrote: post #57
mslot wrote: post #45

Earlier quoted context omitted.

DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus. DuckLake can do things that pg_lake cannot do with Iceberg, and DuckDB can do things Postgres absolutely can't (e.g. query data frames). On the other hand, Postgres can do a lot of things that DuckDB cannot do. For instance, it can handl…

What does data frames mean in this context? I'm used to them in spark or pandas but does this relate to something in how duckDB operates or is it something else?

It's a python data frame.

Re: Pg_lake: Postgres with Iceberg and data lake access

#79

Post by gigatexal »

ozgune wrote: post #2

This is huge! When people ask me what’s missing in the Postgres market, I used to tell them “open source Snowflake.” Crunchy’s Postgres extension is by far the most ahead solution in the market. Huge congrats to Snowflake and the Crunchy team on open sourcing this.

Honestly. Just pay snowflake for the amazing DB and ecosystem it is. And then go build cool stuff unless your value add to customers is infra let them handle all that.
Post reply on HN