Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckLake is an integrated data lake and catalog format

ducklake.select

Post reply on HN
Print view View original

101–110 of 114 posts

Re: DuckLake is an integrated data lake and catalog format

#101

Post by RamtinJ95 »

I love duckDB and this looks just absolutely brilliant!

One question for me is, lets say i want to start using this today and at work we are running snowflake. I get that each analytics person would have to run duckdb + this extension on their local machines and point to the blob store and the database that is running datalake extension, for now that would be say a VM running duckdb. When I run the actual query where does the computation happen? And what if I want a lot of computation?

Is the solution currently to host a huge duckdb VM that everyone ssh's into and run their queries or how does that part work?

Re: DuckLake is an integrated data lake and catalog format

#102

Post by mritchie712 »

RamtinJ95 wrote: post #101

I love duckDB and this looks just absolutely brilliant! One question for me is, lets say i want to start using this today and at work we are running snowflake. I get that each analytics person would have to run duckdb + this extension on their local machines and point to the blob store and the database that is running datalake extension, for now that would be say a VM running duckdb. When I run the actual query where…

the compute would happen on your machine if you were running duckdb locally.

And yes, to get more compute, you'd want to spin up a VM.

What's cool is you can do both (run locally for small stuff, run on VM for heavy workloads).

Re: DuckLake is an integrated data lake and catalog format

#103

Post by amluto »

Dowwie wrote: post #100
amluto wrote: post #50

I have an personal pet peeve about Parquet that is solved, incompatibly, by basically every "data lake / lakehouse" layer on top, and I'd love to see it become compatible: ranged partitioning. I have an application which ought to be a near-perfect match for Parquet. I have a source of timestamped data (basically a time series, except that the intervals might not be evenly spaced -- think log files). A row is a timest…

Time series data is naturally difficult to work with, but avoidable. One solution is to not query raw time series data files. Instead, segment your time series data before you store it, normalizing the timestamps as part of event processing. Sliding window observations will help you find where the event begins and then you adjust the offset until you find where the time series returns to its zero-like position. That'…

Segmenting data is exactly what writing it into non-overlapping Parquet files is. My point is that many tools can read a bucket full of these segments, and most of them can handle a scheme where each file corresponds to a single value of a column, but none of them can agree on how to efficiently segment the data where each segment contains a range, unless a new column is invented for the purpose and all queries add complexity to map onto this type of segmentation.

There’s nothing conceptually or algorithmically difficult about what I want to do. All that’s needed is to encode a range of times into the path of a segment. But Hive didn’t do this, and everyone implemented Hive’s naming scheme, and that’s the status quo now.

Re: DuckLake is an integrated data lake and catalog format

#104

Post by CyberDildonics »

adastra22 wrote: post #15

What is a data lake?

A network file system with a database which can be done with a PC and sqlite, but you need a new term to sell the new thing so now it's data lake and data warehouse and 'blob storage'. How are the 'blobs' stored? Probably with blocks of a constant size linked together to form the the 'full blob'. If this sounds like file systems invented 50 years ago, you just don't understand, the difference is that this is very expensive.

Re: DuckLake is an integrated data lake and catalog format

#105

Post by RamtinJ95 »

mritchie712 wrote: post #102
RamtinJ95 wrote: post #101

I love duckDB and this looks just absolutely brilliant! One question for me is, lets say i want to start using this today and at work we are running snowflake. I get that each analytics person would have to run duckdb + this extension on their local machines and point to the blob store and the database that is running datalake extension, for now that would be say a VM running duckdb. When I run the actual query where…

the compute would happen on your machine if you were running duckdb locally. And yes, to get more compute, you'd want to spin up a VM. What's cool is you can do both (run locally for small stuff, run on VM for heavy workloads).

so in other words we could replace snowflake/bigquery with this solution and get pretty much the same performance?

Re: DuckLake is an integrated data lake and catalog format

#106

Post by Jarwain »

amluto wrote: post #76
bitbang wrote: post #55

Earlier quoted context omitted.

Why is the footer metadata not sufficient for this need? The metadata should contain the min and max timestamp values from the respective column of interest, so that when executing a query, the query tool should be optimizing its query by reading the metadata to determine if that parquet file should be read or not depending on what time range is in the query.

Because the footer metadata is in the Parquet file, which is already far too late to give an efficient query. If I have an S3 bucket containing five years worth of Parquet files, each covering a few days worth of rows, and I tell my favorite query tool (DuckDB, etc) about that bucket, then the tool will need to do a partial read (which is multiple operations, I think, since it will need to find the footer and then re…

Maybe I'm misunderstanding something about how ducklake works, but isn't that the purpose of the 'catalog database'? To store the metadata about all the files to optimize the query?

In theory, going off of the schema diagram they have, all your files are listed in `data_file`, the timestamp range for that file would be in `file_column_stats`, and that information could be used to decide what files to _actually_ read based on your query.

Whether duckdb's query engine takes advantage of this is a different story, but even if it doesn't Yet it should be possible to do so Eventually.

Re: DuckLake is an integrated data lake and catalog format

#107

Post by mritchie712 »

RamtinJ95 wrote: post #105
mritchie712 wrote: post #102

Earlier quoted context omitted.

the compute would happen on your machine if you were running duckdb locally. And yes, to get more compute, you'd want to spin up a VM. What's cool is you can do both (run locally for small stuff, run on VM for heavy workloads).

so in other words we could replace snowflake/bigquery with this solution and get pretty much the same performance?

it depends... but yes, you could likely set it up in a way that matches (or beats) snowflake performance.

Re: DuckLake is an integrated data lake and catalog format

#108

Post by tracnar »

szarnyasg wrote: post #19
formalreconfirm wrote: post #17

Earlier quoted context omitted.

Thank you for your work ! We use DuckDB with dbt-duckdb in production (because on-prem and because we don't need ten thousands nodes) and we love it ! About the COPY statement, it means we can drop Parquet files ourselves in the blob storage ? From my understanding DuckLake was responsible for managing the files on the storage layer.

Great! > About the COPY statement, it means we can drop Parquet files ourselves in the blob storage ? Dropping the Parquet files on the blob storage will not work – you have to COPY them through DuckLake so that the catalog databases is updated with the required catalog and metadata information.

I was also thinking about this use case when reading the announcement. Let's say you have a bunch of parquet files already (on local FS, HTTPS, S3, ...) that you can assume are immutable (or maybe append-only). It would be great if you could attach them to the DuckLake without copying them! From the design doc, it seems it should essentially work, you would read those parquet files to compute the metadata, and insert a reference to the parquet file instead of copying them to the storage you manage. Basically you want to create the catalog independently from the underlying data.

Re: DuckLake is an integrated data lake and catalog format

#109

Post by amluto »

Jarwain wrote: post #106
amluto wrote: post #76

Earlier quoted context omitted.

Because the footer metadata is in the Parquet file, which is already far too late to give an efficient query. If I have an S3 bucket containing five years worth of Parquet files, each covering a few days worth of rows, and I tell my favorite query tool (DuckDB, etc) about that bucket, then the tool will need to do a partial read (which is multiple operations, I think, since it will need to find the footer and then re…

Maybe I'm misunderstanding something about how ducklake works, but isn't that the purpose of the 'catalog database'? To store the metadata about all the files to optimize the query? In theory, going off of the schema diagram they have, all your files are listed in `data_file`, the timestamp range for that file would be in `file_column_stats`, and that information could be used to decide what files to _actually_ read…

Yes, and this is how basically every “lake” thing works. But all the lake solutions add a lot more complexity than just improving the parquet filename scheme, and all of them require that all the readers and all the writers agree on a particular “lake”.

Re: DuckLake is an integrated data lake and catalog format

#110

Post by Jarwain »

amluto wrote: post #109
Jarwain wrote: post #106

Earlier quoted context omitted.

Maybe I'm misunderstanding something about how ducklake works, but isn't that the purpose of the 'catalog database'? To store the metadata about all the files to optimize the query? In theory, going off of the schema diagram they have, all your files are listed in `data_file`, the timestamp range for that file would be in `file_column_stats`, and that information could be used to decide what files to _actually_ read…

Yes, and this is how basically every “lake” thing works. But all the lake solutions add a lot more complexity than just improving the parquet filename scheme, and all of them require that all the readers and all the writers agree on a particular “lake”.

That's fair! I guess I see it as trading technical complexity with the human complexity of getting everyone on board with an update to the standard, and getting that standard implemented across the board. It's a lot easier to get my coworkers to just use duckdb as a reader/writer with ducklake than to change the system.

Frankly, I'm not entirely sure what the process of proposing that change to the hive file scheme would even look like

Post reply on HN