Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckLake is an integrated data lake and catalog format

ducklake.select

Post reply on HN
Print view View original

51–60 of 114 posts

Re: DuckLake is an integrated data lake and catalog format

#51

Post by peterboncz »

data_ders wrote: post #29

the manifesto [1] is the most interesting thing. I agree that DuckDB has the largest potential to disrupt the current order with Iceberg. However, this mostly reads to me as thought experiment: > what if the backend service of an Iceberg catalog was just a SQL database? The manifesto says that maintaining a data lake catalog is easier, which I agree with in theory. s3-files-as-information-schema presents real challen…

https://x.com/peterabcz/status/1927402100922683628

it adds for users the following features to a data lake: - multi-statement & multi-table transactions - SQL views - delta queries - encryption - low latency: no S3 metadata & inlining: store small inserts in-catalog and more!

Re: DuckLake is an integrated data lake and catalog format

#52

Post by mritchie712 »

dkdcio wrote: post #23

This looks awesome. One of my biggest gripe's personally with Iceberg (less-so Delta Lake, but similar) is how difficult it is to just try out on a laptop. Delta Lake has vanilla Python implementations, but those are fragemented and buggy IME. Iceberg has just never worked locally, you need a JVM cluster and a ton of setup. I went down a similar road of trying to use sqlite/postgres+duckdb+parquet files in blob stora…

Here's a step-by-step setup. It's using S3 and RDS, but I wouldn't be hard to swap in a local sqlite instead.

https://www.definite.app/blog/cloud-iceberg-duckdb-aws

Re: DuckLake is an integrated data lake and catalog format

#53

Post by szarnyasg »

TheGuyWhoCodes wrote: post #49

Is there any information about updates to existing rows? The FAQ says "Similarly to other data lakehouse technologies, DuckLake does not support constraints, keys, or indexes." However in Iceberg there are Copy-On-Write and Merge-On-Read strategies dealing with updates.

Yes - updates on existing rows are supported.

(I work at DuckDB Labs.)

Re: DuckLake is an integrated data lake and catalog format

#54

Post by dsp_person »

adastra22 wrote: post #32
szarnyasg wrote: post #16

Earlier quoted context omitted.

The YouTube video “Apache Iceberg: What It Is and Why Everyone’s Talking About It” by Tim Berglund explains data lakes really well in the opening minutes: https://www.youtube.com/watch?v=TsmhRZElPvM

Thanks but I don’t have the time to watch YouTube.

he explains

~40y ago invented data warehouse, where an ETL process overnight would collect data from smaller dbs into a central db (the data warehouse)

~15y ago, data lake (i.e. hadoop) emerged to address scaling and other things. Same idea but ELT instead of ETL: less focus on schema, collect the data into S3 and transform it later

Re: DuckLake is an integrated data lake and catalog format

#55

Post by bitbang »

amluto wrote: post #50

I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning. I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timest…

Why is the footer metadata not sufficient for this need? The metadata should contain the min and max timestamp values from the respective column of interest, so that when executing a query, the query tool should be optimizing its query by reading the metadata to determine if that parquet file should be read or not depending on what time range is in the query.

Re: DuckLake is an integrated data lake and catalog format

#56

Post by dugmartin »

bitbang wrote: post #55
amluto wrote: post #50

I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning. I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timest…

Why is the footer metadata not sufficient for this need? The metadata should contain the min and max timestamp values from the respective column of interest, so that when executing a query, the query tool should be optimizing its query by reading the metadata to determine if that parquet file should be read or not depending on what time range is in the query.

This can also be done using row group metadata within the parquet file. The row group metadata can include the range values of ordinals so you can "partition" on timestamps without having to have a file per time range.

Re: DuckLake is an integrated data lake and catalog format

#57

Post by dkdcio »

frisbm wrote: post #42
dkdcio wrote: post #23

This looks awesome. One of my biggest gripe's personally with Iceberg (less-so Delta Lake, but similar) is how difficult it is to just try out on a laptop. Delta Lake has vanilla Python implementations, but those are fragemented and buggy IME. Iceberg has just never worked locally, you need a JVM cluster and a ton of setup. I went down a similar road of trying to use sqlite/postgres+duckdb+parquet files in blob stora…

Have you tried out PyIceberg yet? It's a pure Python implementation and it works pretty well. It supports a SQL Catalog as well as an In-Memory Catalog via a baked in SQLite SQL Catalog. https://py.iceberg.apache.org/

this looks much simpler than the last time I tried it!

Re: DuckLake is an integrated data lake and catalog format

#59

Post by dkdcio »

wodenokoto wrote: post #46
dkdcio wrote: post #23

This looks awesome. One of my biggest gripe's personally with Iceberg (less-so Delta Lake, but similar) is how difficult it is to just try out on a laptop. Delta Lake has vanilla Python implementations, but those are fragemented and buggy IME. Iceberg has just never worked locally, you need a JVM cluster and a ton of setup. I went down a similar road of trying to use sqlite/postgres+duckdb+parquet files in blob stora…

Delta-io (based on deltalake-r) runs very very easily locally. Just pip install, and write and you get catalog and everything. https://delta-io.github.io/delta-rs/

I tried using it but on more than one occasion hit showstopping bugs -- they're probably fixed by now though

Re: DuckLake is an integrated data lake and catalog format

#60

Post by isoprophlex »

wodenokoto wrote: post #45

I’m building a poor man’s datalake at work, basically putting parquet files in blob storage using deltalake-rs’ python bindings and duck db for querying. However, I constantly run in to problems with concurrent writes. I have a cloud function triggered ever x minutes to pull data from API and that’s fine. But if I need to run a backfill I risk that that process will run at the same time as the timer triggered functio…

Add a randomly chosen suffix to your filenames?
Post reply on HN