Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckLake is an integrated data lake and catalog format

ducklake.select

Post reply on HN
Print view View original

21–30 of 114 posts

Re: DuckLake is an integrated data lake and catalog format

#21

Post by memhole »

formalreconfirm wrote: post #18
spenczar5 wrote: post #14

There is a lot to like here, but once metadata is in the novel Ducklake format, it is hard to picture how you can get good query parallelism, which you need for large datasets. Iceberg already is well supported by lots of heavy-duty query engines and that support is important once you have lots and lots and lots of data.

Someone correct me if I'm wrong but from my understanding, DuckDB will always be the query engine, thus I suppose you will have access to DuckDB query parallelism (single node but multithreaded with disk spilling etc) + statistics-based optimizations like file pruning, predicate pushdown etc offered by DuckLake. I think DuckLake is heavily coupled to DuckDB (Which is good for our use case). Again, this is my understa…

From my perspective the issue is analytics support. You’ll need a step that turns it into something supported by BI tools. Obviously if something like Trino picks up the format it’s not an issue

Re: DuckLake is an integrated data lake and catalog format

#22

Post by prpl »

jakozaur wrote: post #9

Iceberg competitor, addressing some of its shortcomings, like blown-up metadata: https://quesma.com/blog-detail/apache-iceberg-practical-limi... Even Snowflake was using FoundationDB for metadata, whereas Iceberg attempts to use blob storage even for the metadata layer.

metadata bloat can be due to a few things, but it’s manageable.

* number of snapshots

* frequent large schema changes

* lots of small files/row level updates

* lots of stats

The last one IIRC used to be pretty bad especially with larger schemas.

Most engines have ways to help with this - compaction, snapshot exportation, etc… Though it can still be up to the user. S3 tables is supposed to do some of this for you.

If metadata is below 1-5MB it’s really not an issue. Your commit rate is effectively limited by the size of your metadata and the number of writers you have.

I’ve written scripts to fix 1GB+ metadata files in production. Usually it was pruning snapshots without deleting files (relying on bucket policy to later clean things up) or removing old schema versions.

Re: DuckLake is an integrated data lake and catalog format

#23

Post by dkdcio »

This looks awesome. One of my biggest gripe's personally with Iceberg (less-so Delta Lake, but similar) is how difficult it is to just try out on a laptop. Delta Lake has vanilla Python implementations, but those are fragemented and buggy IME. Iceberg has just never worked locally, you need a JVM cluster and a ton of setup. I went down a similar road of trying to use sqlite/postgres+duckdb+parquet files in blob storage, but it was a lot of work.

It seems like this will just work out of the box, and just scale up to very reasonable data sizes. And the work from the DuckDB folks is typically excellent. It's clear they understand this space. Excited to try it out!

Re: DuckLake is an integrated data lake and catalog format

#24

Post by formalreconfirm »

spenczar5 wrote: post #20
formalreconfirm wrote: post #18

Earlier quoted context omitted.

Someone correct me if I'm wrong but from my understanding, DuckDB will always be the query engine, thus I suppose you will have access to DuckDB query parallelism (single node but multithreaded with disk spilling etc) + statistics-based optimizations like file pruning, predicate pushdown etc offered by DuckLake. I think DuckLake is heavily coupled to DuckDB (Which is good for our use case). Again, this is my understa…

I agree with everything you said. I just mean that a single node may be slow when processing those parquet files in a complex aggregation, bottlenecked on network IO or CPU or available memory. If the thesis here is that most datasets are small, fair enough - but then why use a lake instead of big postgres, yknow?

That's the part I don't really get. In the Manifesto they are talking about scaling to hundreds of terabytes and thousands of compute nodes. But DuckDB compute nodes, even if they are very performant, at the end are single nodes, so even if your lakehouse contains TB of data, you will be limited to your biggest client capacity (I know DuckDB works well with data bigger than memory, but still, I suppose it can reach limits at some point). At the end I think DuckLake is aimed at lakehouses of "reasonable" size the same way DuckDB is intended for data of "reasonable" size.

Re: DuckLake is an integrated data lake and catalog format

#25

Post by dkdcio »

formalreconfirm wrote: post #24
spenczar5 wrote: post #20

Earlier quoted context omitted.

I agree with everything you said. I just mean that a single node may be slow when processing those parquet files in a complex aggregation, bottlenecked on network IO or CPU or available memory. If the thesis here is that most datasets are small, fair enough - but then why use a lake instead of big postgres, yknow?

That's the part I don't really get. In the Manifesto they are talking about scaling to hundreds of terabytes and thousands of compute nodes. But DuckDB compute nodes, even if they are very performant, at the end are single nodes, so even if your lakehouse contains TB of data, you will be limited to your biggest client capacity (I know DuckDB works well with data bigger than memory, but still, I suppose it can reach l…

Huge "it depends", but typically organizations are not querying all of their data at once. Usually, they're processing it in some time-based increments.

Even if it's in the TB-range, we're at the point where high-spec laptops can handle it (my own benchmarking: https://ibis-project.org/posts/1tbc/). When I tried to go up to 10TB TPC-H queries on large cloud VMs I did hit some malloc (or other memory) issues, but that was a while ago and I imagine DuckDB can fly past that these days too. Single-node definitely has limits, but it's hard to see how 99%+ of organizations really need distributed computing in 2025.

Re: DuckLake is an integrated data lake and catalog format

#27

Post by buremba »

jakozaur wrote: post #9

Iceberg competitor, addressing some of its shortcomings, like blown-up metadata: https://quesma.com/blog-detail/apache-iceberg-practical-limi... Even Snowflake was using FoundationDB for metadata, whereas Iceberg attempts to use blob storage even for the metadata layer.

I had the same impression but I wouldn't call it competitor after watching their video: https://youtu.be/zeonmOO9jm4?t=4032

They support syncing to Iceberg by writing the manifest and metadata files on demand, and they already have read support for Iceberg. They just fixed Iceberg's core issues but it's not a direct competitor as you can use DuckLake along with Iceberg in a very nice and bidirectional way.

Re: DuckLake is an integrated data lake and catalog format

#28

Post by buremba »

spenczar5 wrote: post #14

There is a lot to like here, but once metadata is in the novel Ducklake format, it is hard to picture how you can get good query parallelism, which you need for large datasets. Iceberg already is well supported by lots of heavy-duty query engines and that support is important once you have lots and lots and lots of data.

You don't need to store the metadata in DuckDB; it can live in your own PostgreSQL/MySQL, similar to Iceberg REST Catalog. They solve query parallelism by allowing you to perform computations on the edge, enabling horizontal scaling the compute layer.

They don't focus on solving the scalability problem in the metadata layer; you might need to scale your PostgreSQL independently as you have many DuckDB compute nodes running on the edge.

Re: DuckLake is an integrated data lake and catalog format

#29

Post by data_ders »

the manifesto [1] is the most interesting thing. I agree that DuckDB has the largest potential to disrupt the current order with Iceberg.

However, this mostly reads to me as thought experiment: > what if the backend service of an Iceberg catalog was just a SQL database?

The manifesto says that maintaining a data lake catalog is easier, which I agree with in theory. s3-files-as-information-schema presents real challenges!

But, what I most want to know is what's the end-user benefit?

What does someone get with this if they're already using Apache Polaris or Lakekeeper as their Iceberg REST catalog?

[1]: https://ducklake.select/manifesto/

Re: DuckLake is an integrated data lake and catalog format

#30

Post by buremba »

My understanding was that MotherDuck was focusing on providing the "multiplayer mode" for DuckDB. It's interesting to see DuckDB Labs supporting data lakes natively. I guess MotherDuck is potentially moving to the UI layer by providing the notebook interface for DuckDB.
Post reply on HN