I wonder how this relates to Mother Duck (https://motherduck.com/)? They do „DuckDB-powered data warehousing“ but predate this substantially.
DuckLake is an integrated data lake and catalog format
31–40 of 114 posts
Re: DuckLake is an integrated data lake and catalog format
#32Re: DuckLake is an integrated data lake and catalog format
#33the manifesto [1] is the most interesting thing. I agree that DuckDB has the largest potential to disrupt the current order with Iceberg. However, this mostly reads to me as thought experiment: > what if the backend service of an Iceberg catalog was just a SQL database? The manifesto says that maintaining a data lake catalog is easier, which I agree with in theory. s3-files-as-information-schema presents real challen…
they say it's faster for one thing - can resolve all metadata in a single query instead of multiple HTTP requests
Re: DuckLake is an integrated data lake and catalog format
#34There is a lot to like here, but once metadata is in the novel Ducklake format, it is hard to picture how you can get good query parallelism, which you need for large datasets. Iceberg already is well supported by lots of heavy-duty query engines and that support is important once you have lots and lots and lots of data.
Someone correct me if I'm wrong but from my understanding, DuckDB will always be the query engine, thus I suppose you will have access to DuckDB query parallelism (single node but multithreaded with disk spilling etc) + statistics-based optimizations like file pruning, predicate pushdown etc offered by DuckLake. I think DuckLake is heavily coupled to DuckDB (Which is good for our use case). Again, this is my understa…
It seems to me that by publishing the spec other non-DuckDB implementations could be built?
It's currently only DuckDB specific because the initial implementation that supports this new catalog is a DuckDB extension
Re: DuckLake is an integrated data lake and catalog format
#35My understanding was that MotherDuck was focusing on providing the "multiplayer mode" for DuckDB. It's interesting to see DuckDB Labs supporting data lakes natively. I guess MotherDuck is potentially moving to the UI layer by providing the notebook interface for DuckDB.
Good point! Anticipating official announcements I can confirm that MotherDuck is indeed intending to both: host DuckLake catalogs, and facilitate querying DuckLakes using DuckDB via its cloud-based DuckDB service.
Re: DuckLake is an integrated data lake and catalog format
#36How do I integrate DuckLake with Apache Spark? Is it a format or a catalog?
Same question for presto, trino, dremio, snowflake, bigquery, etc.
Re: DuckLake is an integrated data lake and catalog format
#37"DuckLake is also able to improve the two biggest performance problems of data lakes: small changes and many concurrent changes."
These I'd argue are not the natural use cases for a data lake, especially a design which uses multiple writers to a given table.
Re: DuckLake is an integrated data lake and catalog format
#38Using SQL as catalog is not new (iceberg supports JDBC catalog from the very beginning).
The main difference is to store metadata and stats also directly in SQL databases, which makes perfect sense for smaller scale data. In fact we were doing something similar in https://github.com/Mooncake-Labs/pg_mooncake, metadata are stored in pg tables and only periodically flush to actual formats like iceberg.
Re: DuckLake is an integrated data lake and catalog format
#39I am a huge fan of what they are doing, particularly putting local compute front and center. However for “BigCorp”, it’s going to be an uphill battle. The incumbents are entrenched and many decision makers will make decisions based on non technical reasons (I.e did my sales exec get me to the F1 Grand Prix).
Re: DuckLake is an integrated data lake and catalog format
#40There is a lot to like here, but once metadata is in the novel Ducklake format, it is hard to picture how you can get good query parallelism, which you need for large datasets. Iceberg already is well supported by lots of heavy-duty query engines and that support is important once you have lots and lots and lots of data.
You don't need to store the metadata in DuckDB; it can live in your own PostgreSQL/MySQL, similar to Iceberg REST Catalog. They solve query parallelism by allowing you to perform computations on the edge, enabling horizontal scaling the compute layer. They don't focus on solving the scalability problem in the metadata layer; you might need to scale your PostgreSQL independently as you have many DuckDB compute nodes r…
Even though it's in your own SQL DB, there's still some sort of layout for the metadata. That's the thing that trino/bigquery/whatever won't understand (yet?).
> They solve query parallelism by allowing you to perform computations on the edge, enabling horizontal scaling the compute layer.
Hmm, I don't understand this one. How do you horizontally scale a query that scans all data to do `select count(*), col from huge_table group by col`, for example? In a traditional map-reduce engine, that turns into parallel execution over chunks of data, which get later merged for the final result. In DuckDB, doesn't that necessarily get done by a single node which has to inspect every row all by itself?