DuckLake is an integrated data lake and catalog format

Post by amluto » Thu, May 29, 2025, 7:02 PM UTC

Jarwain wrote: post #110

amluto wrote: post #109

Earlier quoted context omitted.

Yes, and this is how basically every “lake” thing works. But all the lake solutions add a lot more complexity than just improving the parquet filename scheme, and all of them require that all the readers and all the writers agree on a particular “lake”.

That's fair! I guess I see it as trading technical complexity with the human complexity of getting everyone on board with an update to the standard, and getting that standard implemented across the board. It's a lot easier to get my coworkers to just use duckdb as a reader/writer with ducklake than to change the system. Frankly, I'm not entirely sure what the process of proposing that change to the hive file scheme w…

Post by Jarwain » Sat, May 31, 2025, 1:20 AM UTC

Jarwain wrote: post #112

amluto wrote: post #111

Earlier quoted context omitted.

> Frankly, I'm not entirely sure what the process of proposing that change to the hive file scheme would even look like Maybe convince DuckDB and/or clickhouse-local and/or polars.scan_parquet to implement it as a pilot? If it's a success, other tools might follow suit. Or maybe something like DuckLake could have an option to put column statistics in the filenames. I raised this as a discussion: https://github.com/du…

I'm not super sure about it being in the filename, if only because my understanding is that some of the lakes use it for partitioning and other metadata (metameta-data?). Imo range is probably the most useful statistic in a folder/file name anyways for partitioning purposes. My vote would be for `^` as the range separator to minimize risk of collision and confusion. i.e. `timestamp=2025-03-27T00:00:00-0800^2025-03-30…