Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differential storage: A key building block for a DuckDB-based data warehouse

motherduck.com

Post reply on HN
Print view View original

1–10 of 13 posts

Re: Differential storage: A key building block for a DuckDB-based data warehouse

#2

Post by richieartoul »

This is pretty cool. I'm a bit of a noob when it comes to stuff like FUSE. There is a bunch of commentary in the blog post about how when DuckDB does X, the Differential Storage implementation does Y. If the differential storage implementation just exposes itself as a filesystem though, how does it know what DuckDB is doing at the application layer? For example, how does it know from the filesystem layer that DuckDB is performing a snapshot?

Re: Differential storage: A key building block for a DuckDB-based data warehouse

#6

Post by vegetable26 »

richieartoul wrote: post #2

This is pretty cool. I'm a bit of a noob when it comes to stuff like FUSE. There is a bunch of commentary in the blog post about how when DuckDB does X, the Differential Storage implementation does Y. If the differential storage implementation just exposes itself as a filesystem though, how does it know what DuckDB is doing at the application layer? For example, how does it know from the filesystem layer that DuckDB…

edit: author here

Our current FUSE implementation is specific to DuckDB and does make (as well as validate) some assumptions about DuckDB's access patterns to the underlying files. In this case - we know that currently DuckDB always truncate the WAL on successful checkpoint - which triggers a Differential Storage snapshot under the hood.

We are working on future-proofing this setup by removing our reliance on some of these assumptions and having our FUSE implementation act much more like a generic FS moving forward.

Though in general the truncation of the WAL is still usually a good time to snapshot the database, as the truncation means that the only "state" needed to reconstruct the current database is completely captured in the database file.

Re: Differential storage: A key building block for a DuckDB-based data warehouse

#8

Post by richieartoul »

vegetable26 wrote: post #6
richieartoul wrote: post #2

This is pretty cool. I'm a bit of a noob when it comes to stuff like FUSE. There is a bunch of commentary in the blog post about how when DuckDB does X, the Differential Storage implementation does Y. If the differential storage implementation just exposes itself as a filesystem though, how does it know what DuckDB is doing at the application layer? For example, how does it know from the filesystem layer that DuckDB…

edit: author here Our current FUSE implementation is specific to DuckDB and does make (as well as validate) some assumptions about DuckDB's access patterns to the underlying files. In this case - we know that currently DuckDB always truncate the WAL on successful checkpoint - which triggers a Differential Storage snapshot under the hood. We are working on future-proofing this setup by removing our reliance on some of…

Makes sense, very cool. Thanks!

Re: Differential storage: A key building block for a DuckDB-based data warehouse

#9

Post by vegetable26 »

cmollis wrote: post #4

sounds a bit like what Iceberg does with writes on parquet.

edit: author here

Yah that's a good point. We use a similar copy-on-write implementation as many other storage systems (such as Iceberg) that offer time-travel, branching, etc... where you have snapshots represented by pointers to immutable snapshot-layer or delta files (or however else you want to call them).

In our case, we wanted to provide customers with a very similar/seamless experience of using DuckDB on MotherDuck as they would with local DuckDB. To provide this parity we needed the ability to extend DuckDB's native storage to support these capabilities (time-travel, branching, etc...). This led us to implementing Differential Storage.

Re: Differential storage: A key building block for a DuckDB-based data warehouse

#10

Post by conradev »

I’m curious how their snapshotting and object storage implementation stacks up against SQLite’s offering:

https://sqlite.org/cloudsqlite/doc/trunk/www/index.wiki

SQLite’s object-per-page approach means you can fetch portions of the database as needed from a bucket, whereas the DuckDB snapshotting approach seems to imply that you’d need to fetch the whole database.

Post reply on HN