Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckLake is an integrated data lake and catalog format

ducklake.select

Post reply on HN
Print view View original

71–80 of 114 posts

Re: DuckLake is an integrated data lake and catalog format

#71

Post by akshayka »

dkdcio wrote: post #23

This looks awesome. One of my biggest gripe's personally with Iceberg (less-so Delta Lake, but similar) is how difficult it is to just try out on a laptop. Delta Lake has vanilla Python implementations, but those are fragemented and buggy IME. Iceberg has just never worked locally, you need a JVM cluster and a ton of setup. I went down a similar road of trying to use sqlite/postgres+duckdb+parquet files in blob stora…

It's indeed very easy to try locally! For example in a marimo notebook, just a few lines of code: https://www.youtube.com/watch?v=x6YtqvGcDBY

(Disclosure, I am a developer of marimo.)

Re: DuckLake is an integrated data lake and catalog format

#73

Post by jonstewart »

amluto wrote: post #50

I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning. I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timest…

I think maybe this is a pet peeve of Hive and not of Parquet? Yes, it does require opening the Parquet file to look at the min, max range for the column, but only that data and if the data isn’t in range there shouldn’t be further requests.

That is the kind of metadata that is useful to push up, into something like DuckLake.

Re: DuckLake is an integrated data lake and catalog format

#74

Post by amluto »

dugmartin wrote: post #56
bitbang wrote: post #55

Earlier quoted context omitted.

Why is the footer metadata not sufficient for this need? The metadata should contain the min and max timestamp values from the respective column of interest, so that when executing a query, the query tool should be optimizing its query by reading the metadata to determine if that parquet file should be read or not depending on what time range is in the query.

This can also be done using row group metadata within the parquet file. The row group metadata can include the range values of ordinals so you can "partition" on timestamps without having to have a file per time range.

But I want a file per range! I’m already writing out an entire chunk of rows, and that chunk is a good size for a Parquet file, and that chunk doesn’t overlap the previous chunk.

Sure, metadata in the Parquet file handles this, but a query planner has to read that metadata, whereas a sensible way to stick the metadata in the file path would allow avoiding reading the file at all.

Re: DuckLake is an integrated data lake and catalog format

#75

Post by amluto »

jonstewart wrote: post #73
amluto wrote: post #50

I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning. I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timest…

I think maybe this is a pet peeve of Hive and not of Parquet? Yes, it does require opening the Parquet file to look at the min, max range for the column, but only that data and if the data isn’t in range there shouldn’t be further requests. That is the kind of metadata that is useful to push up, into something like DuckLake.

I guess my peeve could be restated as: Hive’s naming scheme doesn’t handle this, and Parquet per se can’t handle it because it’s out of scope, but all the awesome tools (e.g. DuckDB), when used on Parquet files without something with “lake” or “ice” in the name, use the Hive scheme.

Someone could buck the trend and extend Hive’s scheme to support ranges.

Re: DuckLake is an integrated data lake and catalog format

#76

Post by amluto »

bitbang wrote: post #55
amluto wrote: post #50

I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning. I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timest…

Why is the footer metadata not sufficient for this need? The metadata should contain the min and max timestamp values from the respective column of interest, so that when executing a query, the query tool should be optimizing its query by reading the metadata to determine if that parquet file should be read or not depending on what time range is in the query.

Because the footer metadata is in the Parquet file, which is already far too late to give an efficient query.

If I have an S3 bucket containing five years worth of Parquet files, each covering a few days worth of rows, and I tell my favorite query tool (DuckDB, etc) about that bucket, then the tool will need to do a partial read (which is multiple operations, I think, since it will need to find the footer and then read the footer) of ~500 files just to find out which ones contain the data of interest. A good query plan would be to do a single list operation on the bucket to find the file names and then to read the file or files needed to answer my query.

Iceberg and Delta Lake (I think -- I haven't actually tried it) can do this, but plain Parquet plus Hive partitioning can't, and I'm not aware of any other lightweight scheme that is well supported that can do it. My personal little query tool (which predates Parquet) can do it just fine by the simple expedient of reading directory names.

Re: DuckLake is an integrated data lake and catalog format

#78

Post by buremba »

spenczar5 wrote: post #40
buremba wrote: post #28

Earlier quoted context omitted.

You don't need to store the metadata in DuckDB; it can live in your own PostgreSQL/MySQL, similar to Iceberg REST Catalog. They solve query parallelism by allowing you to perform computations on the edge, enabling horizontal scaling the compute layer. They don't focus on solving the scalability problem in the metadata layer; you might need to scale your PostgreSQL independently as you have many DuckDB compute nodes r…

Even though it's in your own SQL DB, there's still some sort of layout for the metadata. That's the thing that trino/bigquery/whatever won't understand (yet?). > They solve query parallelism by allowing you to perform computations on the edge, enabling horizontal scaling the compute layer. Hmm, I don't understand this one. How do you horizontally scale a query that scans all data to do `select count(*), col from huge…

You are right, you can only scale vertically for single query execution but given that cloud providers now have beefy machines available with TBs of memory and hundreds of CPUs, it’s not a blocker unless you are querying petabyte level raw data. Trino/Spark is still unbeatable in that way but in most cases, you partition the data and use predicate pushdown anyways.

You can scale horizontally if you have many concurrent queries pretty well, that’s what I was referring to.

Re: DuckLake is an integrated data lake and catalog format

#79

Post by mrbungie »

memhole wrote: post #21
formalreconfirm wrote: post #18

Earlier quoted context omitted.

Someone correct me if I'm wrong but from my understanding, DuckDB will always be the query engine, thus I suppose you will have access to DuckDB query parallelism (single node but multithreaded with disk spilling etc) + statistics-based optimizations like file pruning, predicate pushdown etc offered by DuckLake. I think DuckLake is heavily coupled to DuckDB (Which is good for our use case). Again, this is my understa…

From my perspective the issue is analytics support. You’ll need a step that turns it into something supported by BI tools. Obviously if something like Trino picks up the format it’s not an issue

DuckDB has support for ODBC/ADBC and even JDBC (so Trino should be easy, https://trino.io/docs/current/connector/duckdb.html).

Re: DuckLake is an integrated data lake and catalog format

#80

Post by ryanworl »

tishj wrote: post #62
peterboncz wrote: post #51

Earlier quoted context omitted.

https://x.com/peterabcz/status/1927402100922683628 it adds for users the following features to a data lake: - multi-statement & multi-table transactions - SQL views - delta queries - encryption - low latency: no S3 metadata & inlining: store small inserts in-catalog and more!

One thing to add to this: Snapshots can be retained (though rewritten) even through compaction As a consequence of compaction, when deleting the build up of many small add/delete files, in a format like Iceberg, you would lose the ability to time travel to those earlier states. With DuckLake's ability to refer to parts of parquet files, we can preserve the ability to time travel, even after deleting the old parquet f…

Does this trick preclude the ability to sort your data within a partition? You wouldn’t be able to rely on the row IDs being sequential anymore to be able to just refer to a prefix of them within a newly created file.
Post reply on HN