Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckLake is an integrated data lake and catalog format

ducklake.select

Post reply on HN
Print view View original

61–70 of 114 posts

Re: DuckLake is an integrated data lake and catalog format

#61

Post by TheGuyWhoCodes »

szarnyasg wrote: post #53
TheGuyWhoCodes wrote: post #49

Is there any information about updates to existing rows? The FAQ says "Similarly to other data lakehouse technologies, DuckLake does not support constraints, keys, or indexes." However in Iceberg there are Copy-On-Write and Merge-On-Read strategies dealing with updates.

Yes - updates on existing rows are supported. (I work at DuckDB Labs.)

Thanks szarnyasg. If I've got you here, can you use the ducklake extension commands to get the parquet files for a query without running said query?

That way you could use another query engine while still use duckdb to handle the data mutation.

Re: DuckLake is an integrated data lake and catalog format

#62

Post by tishj »

peterboncz wrote: post #51
data_ders wrote: post #29

the manifesto [1] is the most interesting thing. I agree that DuckDB has the largest potential to disrupt the current order with Iceberg. However, this mostly reads to me as thought experiment: > what if the backend service of an Iceberg catalog was just a SQL database? The manifesto says that maintaining a data lake catalog is easier, which I agree with in theory. s3-files-as-information-schema presents real challen…

https://x.com/peterabcz/status/1927402100922683628 it adds for users the following features to a data lake: - multi-statement & multi-table transactions - SQL views - delta queries - encryption - low latency: no S3 metadata & inlining: store small inserts in-catalog and more!

One thing to add to this: Snapshots can be retained (though rewritten) even through compaction

As a consequence of compaction, when deleting the build up of many small add/delete files, in a format like Iceberg, you would lose the ability to time travel to those earlier states.

With DuckLake's ability to refer to parts of parquet files, we can preserve the ability to time travel, even after deleting the old parquet files

Re: DuckLake is an integrated data lake and catalog format

#63

Post by [deleted] »

dugmartin wrote: post #56
bitbang wrote: post #55

Earlier quoted context omitted.

Why is the footer metadata not sufficient for this need? The metadata should contain the min and max timestamp values from the respective column of interest, so that when executing a query, the query tool should be optimizing its query by reading the metadata to determine if that parquet file should be read or not depending on what time range is in the query.

This can also be done using row group metadata within the parquet file. The row group metadata can include the range values of ordinals so you can "partition" on timestamps without having to have a file per time range.

[deleted]

Re: DuckLake is an integrated data lake and catalog format

#65

Post by hendiatris »

amluto wrote: post #50

I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning. I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timest…

In the lower level arrow/parquet libraries you can control the row groups, and even the data pages (although it’s a lot more work). I have used this heavily with the arrow-rs crate to drastically improve (like 10x) how quickly data could be queried from files. Some row groups will have just a few rows, others will have thousands, but being able to bypass searching in many row groups makes the skew irrelevant.

Just beware that one issue you can have is the limit of row groups per file (2^15).

Re: DuckLake is an integrated data lake and catalog format

#66

Post by adastra22 »

dsp_person wrote: post #54
adastra22 wrote: post #32

Earlier quoted context omitted.

Thanks but I don’t have the time to watch YouTube.

he explains ~40y ago invented data warehouse, where an ETL process overnight would collect data from smaller dbs into a central db (the data warehouse) ~15y ago, data lake (i.e. hadoop) emerged to address scaling and other things. Same idea but ELT instead of ETL: less focus on schema, collect the data into S3 and transform it later

Thank you!

Re: DuckLake is an integrated data lake and catalog format

#68

Post by simlevesque »

dugmartin wrote: post #56
bitbang wrote: post #55

Earlier quoted context omitted.

Why is the footer metadata not sufficient for this need? The metadata should contain the min and max timestamp values from the respective column of interest, so that when executing a query, the query tool should be optimizing its query by reading the metadata to determine if that parquet file should be read or not depending on what time range is in the query.

This can also be done using row group metadata within the parquet file. The row group metadata can include the range values of ordinals so you can "partition" on timestamps without having to have a file per time range.

I wish we had more control of the row group metadata when writing Parquet files with DuckDB.

Re: DuckLake is an integrated data lake and catalog format

#70

Post by snthpy »

iampims wrote: post #5

Great idea, poor naming. If you’re aiming for a standard of sorts, tying it to a specific software by reusing its name feels counter productive. “Ducklake DuckDB extension” really rolls off the tongue /s.

True. The format looks really open so it would be better to have a more independent name. DuckLake for the DuckDB extension name is great in my opinion but for the table format something like SQLake or AcidLake might be more apt. The latter doesn't sound very appealing though, probably especially for ducks.
Post reply on HN