Earlier quoted context omitted.
Content-addressed backups sound something like how git stores data, is that the best way to think about them? And if so, what would be the main differences between just committing to a git repo for example?
The "rolling window hashes" from the comment suggests sub-file matching at any offset. (See Bently-McIlroy diff algo/how rsync efficiently finds matches, for example.) I'm not aware that git performs this sort of deduplication. Better yet would be to use a rolling hash to decide where to cut the blocks, and then use a locality-aware hash (SimHash, etc.) to find similar blocks. Perform a topological sort to decide whi…
Here's a demo.
First, create two test files. The files both contain the same two 1-megabyte chunks of random bytes, but in the opposite order:
$ openssl rand 1000000 > a
$ openssl rand 1000000 > b
$ cat a b > ab
$ cat b a > ba
$ du -sh ab ba
2.0M ab
2.0M ba
$ git init
Initialized empty Git repository in /tmp/x/.git/
$ git add ab ba
$ git commit -m 'add two files'
[master (root-commit) 7f75af0] add two files
2 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 ab
create mode 100644 ba
$ du -sh .git
4.0M .git
$ git gc
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 6 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), done.
Total 4 (delta 1), reused 0 (delta 0), pack-reused 0
$ du -sh .git
2.1M .git