Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckLake is an integrated data lake and catalog format

ducklake.select

Post reply on HN
Print view View original

91–100 of 114 posts

Re: DuckLake is an integrated data lake and catalog format

#91

Post by raihansaputra »

nehalem wrote: post #31

I wonder how this relates to Mother Duck ( https://motherduck.com/ )? They do „DuckDB-powered data warehousing“ but predate this substantially.

i think a way to see it is MotherDuck is a service to just throw your data at at they will sort it (using duckdb underneath) and you can use DuckDB to iterface with your data. But if you want to be more "lakehouse" or maybe down the line there are more integrations with DuckLake ir you want data to be stored in a blob storage, you can use DuckLake with MotherDuck as the metadata store.

Re: DuckLake is an integrated data lake and catalog format

#92

Post by amluto »

mrlongroots wrote: post #89
amluto wrote: post #74

Earlier quoted context omitted.

But I want a file per range! I’m already writing out an entire chunk of rows, and that chunk is a good size for a Parquet file, and that chunk doesn’t overlap the previous chunk. Sure, metadata in the Parquet file handles this, but a query planner has to read that metadata, whereas a sensible way to stick the metadata in the file path would allow avoiding reading the file at all.

I have the same gripe. You want a canonical standard that's like "hive partitioning" but defines the range [val1, val2) as column=val1_val2. It's a trivial addition on top of Parquet.

That would do the trick, as would any other spelling of the same thing.

Re: DuckLake is an integrated data lake and catalog format

#93

Post by raihansaputra »

formalreconfirm wrote: post #10
rtyu1120 wrote: post #8

Earlier quoted context omitted.

Quite a bummer, particularly because the main selling point is that it can be utilized with any SQL database (iiuc).

If I understand the Manifesto correctly, the metadata db can be any SQL database but the client needs to be DuckDB + DuckLake extension no ?

*for now. The principle in the client side (especially read only) should be the same with Iceberg. Ideally there's an Iceberg adapter for clients.

Re: DuckLake is an integrated data lake and catalog format

#94

Post by ivovk »

jamesblonde wrote: post #72

How will DuckLake work with other Iceberg clients - like Python (Polars), Spark, Flink, etc? Do you need to put a REST API in front of it this duckdb instance to make it an Iceberg Catalog?

My understanding is that DuckLake, while being open source format, is not compatible with Iceberg, since it addressees some of it’s shortcomings, such as metadata stored in blob storage.

Re: DuckLake is an integrated data lake and catalog format

#95

Post by teleforce »

nojvek wrote: post #43
nehalem wrote: post #31

I wonder how this relates to Mother Duck ( https://motherduck.com/ )? They do „DuckDB-powered data warehousing“ but predate this substantially.

Motherduck is hosting duckdb in cloud. DuckLake is a much more open system. Ducklake you can build petabyte scale warehouse with multiple readers and writer instances, all transactional on your s3, on your ec2 instances. Motherduck has limitations like only one writer instance. Read replicas can be 1m behind (not transactional). Having different instances concurrently writing to different tables is not possible. Duck…

Just wondering does DuckLake utilizing Open Table Formats (OTFs) since I don't see it's mentioned anywhere in the website?

Re: DuckLake is an integrated data lake and catalog format

#96

Post by wodenokoto »

isoprophlex wrote: post #60
wodenokoto wrote: post #45

I’m building a poor man’s datalake at work, basically putting parquet files in blob storage using deltalake-rs’ python bindings and duck db for querying. However, I constantly run in to problems with concurrent writes. I have a cloud function triggered ever x minutes to pull data from API and that’s fine. But if I need to run a backfill I risk that that process will run at the same time as the timer triggered functio…

Add a randomly chosen suffix to your filenames?

That doesn’t change the manifest, which keeps tabs of which rows are current and which are soft deleted, making time travel possible.

Re: DuckLake is an integrated data lake and catalog format

#97

Post by wodenokoto »

ed_elliott_asc wrote: post #81
wodenokoto wrote: post #45

I’m building a poor man’s datalake at work, basically putting parquet files in blob storage using deltalake-rs’ python bindings and duck db for querying. However, I constantly run in to problems with concurrent writes. I have a cloud function triggered ever x minutes to pull data from API and that’s fine. But if I need to run a backfill I risk that that process will run at the same time as the timer triggered functio…

Take a lease on the json file before you attempt the write and queue writes that way

What does the worker that tries to commit do when the json manifest is locked? Wait and try again?

Re: DuckLake is an integrated data lake and catalog format

#98

Post by Dowwie »

Hopefully this clarifies the value proposition for others:

Existing Lakehouse systems like Iceberg store crucial table information (like schema and file lists) as many small "metadata files" in cloud object storage (like S3). Accessing these files requires numerous network calls, making operations like query planning and updating tables inefficient and prone to conflicts. DuckLake solves this by putting all that metadata into a fast, transactional SQL database, using a single query to get what's needed, which is much quicker and more reliable.

Re: DuckLake is an integrated data lake and catalog format

#99

Post by raihansaputra »

teleforce wrote: post #95
nojvek wrote: post #43

Earlier quoted context omitted.

Motherduck is hosting duckdb in cloud. DuckLake is a much more open system. Ducklake you can build petabyte scale warehouse with multiple readers and writer instances, all transactional on your s3, on your ec2 instances. Motherduck has limitations like only one writer instance. Read replicas can be 1m behind (not transactional). Having different instances concurrently writing to different tables is not possible. Duck…

Just wondering does DuckLake utilizing Open Table Formats (OTFs) since I don't see it's mentioned anywhere in the website?

No. DuckLake is implementing the Open Table Format (and the Catalog above the Table Format). Not utilizing them, but an alternate implementation.

Re: DuckLake is an integrated data lake and catalog format

#100

Post by Dowwie »

amluto wrote: post #50

I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning. I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timest…

Time series data is naturally difficult to work with, but avoidable. One solution is to not query raw time series data files. Instead, segment your time series data before you store it, normalizing the timestamps as part of event processing. Sliding window observations will help you find where the event begins and then you adjust the offset until you find where the time series returns to its zero-like position. That's your event.
Post reply on HN