[deleted]
Differential storage: A key building block for a DuckDB-based data warehouse
11–13 of 13 posts
Re: Differential storage: A key building block for a DuckDB-based data warehouse
#12I did something similar when I worked on an internal timeseries DB with optimization for long term and short term storage.
The EFS thing is a bit tricky. Once your credits are done it's super slow, so I bet they do provisioned but that's expensive.
The other thing was that multiple instances trying to reach the same EFS endpoint at the same time ,for the first time, will fail so you need to stagger it or retry and the docker mount plugin was a bit iffy.
That said cool concept.
Re: Differential storage: A key building block for a DuckDB-based data warehouse
#13This is pretty cool. I'm a bit of a noob when it comes to stuff like FUSE. There is a bunch of commentary in the blog post about how when DuckDB does X, the Differential Storage implementation does Y. If the differential storage implementation just exposes itself as a filesystem though, how does it know what DuckDB is doing at the application layer? For example, how does it know from the filesystem layer that DuckDB…
edit: author here Our current FUSE implementation is specific to DuckDB and does make (as well as validate) some assumptions about DuckDB's access patterns to the underlying files. In this case - we know that currently DuckDB always truncate the WAL on successful checkpoint - which triggers a Differential Storage snapshot under the hood. We are working on future-proofing this setup by removing our reliance on some of…
I'm not too familiar with FUSE, but I would imagine that you are doing something similar to registering a custom filesystem on duckdb, then intercepting certain filesystem activities to trigger all the magic described in the blog?
Also, do you think any of this is going to make its way back into the duckdb core, or perhaps even influencing the duckdb developers to make some of this native or easier (avoiding assumptions about what duckdb is doing)? Perhaps some kind of trigger on checkpoint/similar activities?
And btw, very interesting to read this announcement after reading through the S3 discussion yesterday.