Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pg_lake: Postgres with Iceberg and data lake access

github.com

Post reply on HN
Print view View original

121–126 of 126 posts

Re: Pg_lake: Postgres with Iceberg and data lake access

#121

Post by qaq »

andiz wrote: post #119
qaq wrote: post #60

Earlier quoted context omitted.

even partition elimination is pretty primitive. For Query optimizer Iceberg is really not a primary target. The overall interaction with even technical people gives strong this is a sales org that happens to own an OLAP db product vibe.

I have to very much disagree on that. All pruning techniques in Snowflake work equally well both on their proprietary format as well for Iceberg tables. Iceberg is nowadays a first-class citizen in Snowflake, with pruning working at the file level, row group level, and page level. Same is true for other query optimization techniques. There is even a paper on that: https://arxiv.org/abs/2504.11540 Where pruning differ…

sure select distinct customer_id ... customer_id is first part of partition key you really don't need to do a tablescan to resolve that do you ?

Re: Pg_lake: Postgres with Iceberg and data lake access

#122

Post by [deleted] »

mslot wrote: post #58
jabr wrote: post #48

Earlier quoted context omitted.

How does this compare to https://www.mooncake.dev/pgmooncake ? It seems there are several projects like this now, with each taking a slightly different approach optimized for different use cases?

Definitely similar goals, from the Mooncake author: https://news.ycombinator.com/item?id=43298145 I think pg_mooncake is still relatively early stage. There's a degree of maturity to pg_lake resulting from our team's experience working on extensions like Citus, pg_documentdb, pg_cron, and many others in the past. For instance, in pg_lake all SQL features and transactions just work, the hybrid query engine can delegat…

[deleted]

Re: Pg_lake: Postgres with Iceberg and data lake access

#123

Post by enether »

nojvek wrote: post #110
gigatexal wrote: post #91

Earlier quoted context omitted.

Locked in? I mean they’re your partner. As long as you’re deriving value from them the partnership is still valuable no?

Not all vendors are same. Snowflake charges an arm and leg for compute. It’s 36x more expensive than equivalent EC2 compute.

yeah, this exchange reads like a sales ad

Re: Pg_lake: Postgres with Iceberg and data lake access

#124

Post by enether »

wodenokoto wrote: post #93
fifilura wrote: post #86

How do you use your data lake? For me it is much more than just storing data, it is just as much for crunching numbers in unpredictable ways. And this is where postgres does not cut it. You need some more CPU and RAM than what you pay for in your postgres instance. I.e. a distributed engine where you don't have to worry about how big your database instance is today.

The point about a datalake is to separate computer and storage. Postgres isn’t a compute layer it’s an access layer. Your compute asks Postgres “what is the current data for these keys?” Or “what was the current data as of two weeks ago for these keys?” And your compute will then download and aggregate your analytics query directly from the parquet files.

but most serious compute engines already speak Iceberg, what do they gain from interfacing with PG now?

My understanding is the opposite - PG cuts it as a compute layer for small amounts of data, and this is where it excels.

I also assume `pg_lake` was built mainly with the intention of creating/writing tables, and the ability to read comes "for free" as an extra, since Iceberg integration is already written.

Re: Pg_lake: Postgres with Iceberg and data lake access

#125

Post by enether »

pgguru wrote: post #32
spenczar5 wrote: post #29

Earlier quoted context omitted.

I think I don't understand postgres enough, so forgive this naive question, but what does pushing down to the remote tables mean? Does it allow parallelism? If I query a very large iceberg table, will this system fan the work out to multiple duckdb executors and gather the results back in?

In any query engine you can execute the same query in different ways. The more restrictions that you can apply on the DuckDB side the less data you need to return to Postgres. For instance, you could compute a `SELECT COUNT(*) FROM mytable WHERE first_name = 'David'` by querying all the rows from `mytable` on the DuckDB side, returning all the rows, and letting Postgres itself count the number of results, but this is…

What does "remotely" mean in this context? My understanding is that all of this runs on the same machine - your Postgres server machine runs DuckDB on the same machine via the extension.

I assume you simply mean DuckDB, being a columnar engine, is more efficient in doing this work than PG is

Re: Pg_lake: Postgres with Iceberg and data lake access

#126

Post by enether »

Do I understand it correctly that DuckDB would run embedded on the machine running Postgres (i.e through the extension), and this limits query processing ability to whatever that machine can comfortably handle?

What are the deployment implications if one wants to integrate this in production? Surely they'd need a much larger Postgres machine at a minimum.

Is there concern re: "hot neighbour" problems if the DuckDB queries get too heavy? How is that sort of issue potentially handled? I understood from another query that DuckDB is ran in a separate process, so there is room to potentially throttle it

Post reply on HN