Not exactly sure what it's for? it's to stream your data to Parquet files on (eg) S3 and keep somewhere the exact schema at each point in time? or is it something else?
would be nice to have some tutorial/use-cases in the doc :)
DuckLake is an integrated data lake and catalog format
41–50 of 114 posts
Re: DuckLake is an integrated data lake and catalog format
#42This looks awesome. One of my biggest gripe's personally with Iceberg (less-so Delta Lake, but similar) is how difficult it is to just try out on a laptop. Delta Lake has vanilla Python implementations, but those are fragemented and buggy IME. Iceberg has just never worked locally, you need a JVM cluster and a ton of setup. I went down a similar road of trying to use sqlite/postgres+duckdb+parquet files in blob stora…
Have you tried out PyIceberg yet? It's a pure Python implementation and it works pretty well. It supports a SQL Catalog as well as an In-Memory Catalog via a baked in SQLite SQL Catalog.
Re: DuckLake is an integrated data lake and catalog format
#43I wonder how this relates to Mother Duck ( https://motherduck.com/ )? They do „DuckDB-powered data warehousing“ but predate this substantially.
Motherduck is hosting duckdb in cloud. DuckLake is a much more open system.
Ducklake you can build petabyte scale warehouse with multiple readers and writer instances, all transactional on your s3, on your ec2 instances.
Motherduck has limitations like only one writer instance. Read replicas can be 1m behind (not transactional).
Having different instances concurrently writing to different tables is not possible.
Ducklake gives proper separation of compute and storage with a transactional metadata layer.
Re: DuckLake is an integrated data lake and catalog format
#44Earlier quoted context omitted.
You don't need to store the metadata in DuckDB; it can live in your own PostgreSQL/MySQL, similar to Iceberg REST Catalog. They solve query parallelism by allowing you to perform computations on the edge, enabling horizontal scaling the compute layer. They don't focus on solving the scalability problem in the metadata layer; you might need to scale your PostgreSQL independently as you have many DuckDB compute nodes r…
Even though it's in your own SQL DB, there's still some sort of layout for the metadata. That's the thing that trino/bigquery/whatever won't understand (yet?). > They solve query parallelism by allowing you to perform computations on the edge, enabling horizontal scaling the compute layer. Hmm, I don't understand this one. How do you horizontally scale a query that scans all data to do `select count(*), col from huge…
you're correct that duckdb doesn't do any multi-node map-reduce, however duckdb utilizes all available cores on a node quite effectively to parallelize scanning. And node sizes nowadays get upto 192 vCPUs.
A single node can scan through several gigabytes of data per second. When the column data is compressed through various algorithms, this means billions of rows / sec.
Re: DuckLake is an integrated data lake and catalog format
#45I’m building a poor man’s datalake at work, basically putting parquet files in blob storage using deltalake-rs’ python bindings and duck db for querying.
However, I constantly run in to problems with concurrent writes. I have a cloud function triggered ever x minutes to pull data from API and that’s fine.
But if I need to run a backfill I risk that that process will run at the same time as the timer triggered function. Especially if I load my backfill queue with hundreds of runs that needs to be pulled and they start saturating the workers in the cloud function.
Re: DuckLake is an integrated data lake and catalog format
#46This looks awesome. One of my biggest gripe's personally with Iceberg (less-so Delta Lake, but similar) is how difficult it is to just try out on a laptop. Delta Lake has vanilla Python implementations, but those are fragemented and buggy IME. Iceberg has just never worked locally, you need a JVM cluster and a ton of setup. I went down a similar road of trying to use sqlite/postgres+duckdb+parquet files in blob stora…
Delta-io (based on deltalake-r) runs very very easily locally. Just pip install, and write and you get catalog and everything.
Re: DuckLake is an integrated data lake and catalog format
#47Earlier quoted context omitted.
The YouTube video “Apache Iceberg: What It Is and Why Everyone’s Talking About It” by Tim Berglund explains data lakes really well in the opening minutes: https://www.youtube.com/watch?v=TsmhRZElPvM
Thanks but I don’t have the time to watch YouTube.
It's your db but on s3.
Re: DuckLake is an integrated data lake and catalog format
#48I wonder how this relates to Mother Duck ( https://motherduck.com/ )? They do „DuckDB-powered data warehousing“ but predate this substantially.
For what it's worth, MotherDuck and DuckLake will play together very nicely. You will be able to have your MotherDuck data stored in DuckLake, improving scalability, concurrency, and consistency while also giving access to the underlying data to third-party tools. We've been working on this for the last couple of months, and will share more soon.
Re: DuckLake is an integrated data lake and catalog format
#49Is there any information about updates to existing rows?
The FAQ says "Similarly to other data lakehouse technologies, DuckLake does not support constraints, keys, or indexes."
However in Iceberg there are Copy-On-Write and Merge-On-Read strategies dealing with updates.
Re: DuckLake is an integrated data lake and catalog format
#50I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning.
I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timestamp and a bunch of other columns, and all the columns have data types that Parquet handles just fine [0]. The data accumulates, and it's written out in batches, and the batches all have civilized sizes. The data is naturally partitioned on some partition column, and there is only one writer for each value of the partition column. So far, so good -- the operation of writing a batch is a single file creation or create call to any object store. The partition column maps to the de-facto sort-of-standard Hive partitioning scheme.
Except that the data is (obviously) also partitioned on the timestamp -- each batch covers a non-overlapping range of timestamps. And Hive partitioning can't represent this. So none of the otherwise excellent query tools can naturally import the data unless I engage in a gross hack:
I could also partition on a silly column like "date". This involves aligning batches to date boundaries and also makes queries uglier.
I could just write the files and import ".parquet". This kills performance and costs lots of money.
I could use Iceberg or Delta Lake or whatever for the sole benefit that their client tools can handle ranged partitions. Gee thanks. I don't actually need any of the other complexity.
It would IMO be really really nice if everyone could come up with a directory-name or filename scheme for ranged partitioning.
[0] My other peeve is that a Parquet row and an Arrow row and a Thrift message and a protobuf message, etc, are almost* but not quite the same thing. It would be awesome if there was a companion binary format for a single Parquet row or a stream of rows so that tools could cooperate more easily on producing the data that eventually gets written into Parquet files.