Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pg_lake: Postgres with Iceberg and data lake access

github.com

Post reply on HN
Print view View original

51–60 of 126 posts

Re: Pg_lake: Postgres with Iceberg and data lake access

#51

Post by j_kao »

jabr wrote: post #48
mslot wrote: post #45

Earlier quoted context omitted.

DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus. DuckLake can do things that pg_lake cannot do with Iceberg, and DuckDB can do things Postgres absolutely can't (e.g. query data frames). On the other hand, Postgres can do a lot of things that DuckDB cannot do. For instance, it can handl…

How does this compare to https://www.mooncake.dev/pgmooncake ? It seems there are several projects like this now, with each taking a slightly different approach optimized for different use cases?

FYI the mooncake team was acquired by Databricks so it's basically vendors trying to compete on features now :)

Re: Pg_lake: Postgres with Iceberg and data lake access

#55

Post by ozkatz »

qaq wrote: post #49
barrrrald wrote: post #27

Earlier quoted context omitted.

One thing I admire about Snowflake is a real commitment to self-cannibalization. They were super out front with Iceberg even though it could disrupt them, because that's what customers were asking for and they're willing to bet they'll figure out how to make money in that new world Video of their SVP of Product talking about it here: https://youtu.be/PERZMGLhnF8?si=DjS_OgbNeDpvLA04&t=1195

Have you interacted with Snowflake teams much? We are using external iceberg tables with snowflake. Every interaction pretty much boils down to you really should not be using iceberg you should be using snowflake for storage. It's also pretty obvious some things are strategically not implemented to push you very strongly in that direction.

Out of curiosity - can you share a few examples of functionality currently not supported with Iceberg but that works well with their internal format?

Re: Pg_lake: Postgres with Iceberg and data lake access

#56

Post by taude »

apexalpha wrote: post #54

I’m not super into the Data sphere but my company relies heavily on Snowflake which is becoming an issue. This announcement seems huge to me, no?! Is this really an open source Snowflake covering most use cases?

there's also plenty of other options for warehouse/compute processing of iceberg data storage.

I think this is a pretty big deal, though.

Snowflake does a lot more, though, especially around sharing data across company boundaries.

Re: Pg_lake: Postgres with Iceberg and data lake access

#57

Post by anktor »

mslot wrote: post #45
boshomi wrote: post #36

Why not just use Ducklake?[1] That reduces complexity[2] since only DuckDB and PostgreSQL with pg_duckdb are required. [1] https://ducklake.select/ [2] DuckLake - The SQL-Powered Lakehouse Format for the Rest of Us by Prof. Hannes Mühleisen: https://www.youtube.com/watch?v=YQEUkFWa69o

DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus. DuckLake can do things that pg_lake cannot do with Iceberg, and DuckDB can do things Postgres absolutely can't (e.g. query data frames). On the other hand, Postgres can do a lot of things that DuckDB cannot do. For instance, it can handl…

What does data frames mean in this context? I'm used to them in spark or pandas but does this relate to something in how duckDB operates or is it something else?

Re: Pg_lake: Postgres with Iceberg and data lake access

#58

Post by mslot »

jabr wrote: post #48
mslot wrote: post #45

Earlier quoted context omitted.

DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus. DuckLake can do things that pg_lake cannot do with Iceberg, and DuckDB can do things Postgres absolutely can't (e.g. query data frames). On the other hand, Postgres can do a lot of things that DuckDB cannot do. For instance, it can handl…

How does this compare to https://www.mooncake.dev/pgmooncake ? It seems there are several projects like this now, with each taking a slightly different approach optimized for different use cases?

Definitely similar goals, from the Mooncake author: https://news.ycombinator.com/item?id=43298145

I think pg_mooncake is still relatively early stage.

There's a degree of maturity to pg_lake resulting from our team's experience working on extensions like Citus, pg_documentdb, pg_cron, and many others in the past.

For instance, in pg_lake all SQL features and transactions just work, the hybrid query engine can delegate different fragments of the query into DuckDB if the whole query cannot be handled, and having a robust DuckDB integration with a single DuckDB instance (rather than 1 per session) in a separate server process helps make it production-ready. It is used in heavy production workloads already.

No compromise on Postgres features is especially hard to achieve, but after a decade of trying to get there with Citus, we knew we had to get that right from day 1.

Basically, we could speed run this thing into a comprehensive, production-ready solution. I think others will catch up, but we're not sitting still either. :)

Re: Pg_lake: Postgres with Iceberg and data lake access

#59

Post by mritchie712 »

mslot wrote: post #45
boshomi wrote: post #36

Why not just use Ducklake?[1] That reduces complexity[2] since only DuckDB and PostgreSQL with pg_duckdb are required. [1] https://ducklake.select/ [2] DuckLake - The SQL-Powered Lakehouse Format for the Rest of Us by Prof. Hannes Mühleisen: https://www.youtube.com/watch?v=YQEUkFWa69o

DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus. DuckLake can do things that pg_lake cannot do with Iceberg, and DuckDB can do things Postgres absolutely can't (e.g. query data frames). On the other hand, Postgres can do a lot of things that DuckDB cannot do. For instance, it can handl…

> For instance, it can handle >100k single row inserts/sec.

DuckLake already has data-inlining for the DuckDB catalog, seems this will be possible once it's supported in the pg catalog.

> Postgres also has a more natural persistence & continuous processing story, so you can set up pg_cron jobs and use PL/pgSQL (with heap tables for bookkeeping) to do orchestration.

This is true, but it's not clear where I'd use this in practice. e.g. if I need to run a complex ETL job, I probably wouldn't do it in pg_cron.

Re: Pg_lake: Postgres with Iceberg and data lake access

#60

Post by qaq »

ozkatz wrote: post #55
qaq wrote: post #49

Earlier quoted context omitted.

Have you interacted with Snowflake teams much? We are using external iceberg tables with snowflake. Every interaction pretty much boils down to you really should not be using iceberg you should be using snowflake for storage. It's also pretty obvious some things are strategically not implemented to push you very strongly in that direction.

Out of curiosity - can you share a few examples of functionality currently not supported with Iceberg but that works well with their internal format?

even partition elimination is pretty primitive. For Query optimizer Iceberg is really not a primary target. The overall interaction with even technical people gives strong this is a sales org that happens to own an OLAP db product vibe.
Post reply on HN