OpenZFS deduplication is good now and you shouldn't use it
despairlabs.com
OpenZFS deduplication is good now and you shouldn't use it
1–10 of 250 posts
Re: OpenZFS deduplication is good now and you shouldn't use it
#2I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools compress within a crawl to create wacz or warc.gz files, but I don't think anyone has tried to do compression across the entire database of all snapshots ever taken by a tool.
Big thank you to all the people that worked on it!
BTW has anyone tried a probabilistic dedup approach using soemthing like a bloom filter so you don't have to store the entire dedup table of hashes verbatim? Collect groups of ~100 block hashes into a bucket each, and store a hyper compressed representation in a bloom filter. On write, lookup the hash of the block to write in the bloom filter, and if a potential dedup hit is detected, walk the 100 blocks in the matching bucket manually to look for any identical hashes. In theory you could do this with layers of bloom filters with different resolutions and dynamically swap out the heavier ones to disk when memory pressure is too high to keep the high resolution ones in RAM. Allowing the accuracy of the bloom filter to be changed as a tunable parameter would let people choose their preference around CPU time/overhead:bytes saved ratio.
Re: OpenZFS deduplication is good now and you shouldn't use it
#3[deleted]
Re: OpenZFS deduplication is good now and you shouldn't use it
#4I wonder why they are having so much trouble getting this working properly with smaller RAM footprints. We have been using commercial storage appliances that have been able to do this for about a decade (at least) now, even on systems with "little" RAM (compared to the amount of disk storage attached).
Just store fingerprints in a database and run through that at night and fixup the block pointers...
Re: OpenZFS deduplication is good now and you shouldn't use it
#5I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately.
Because:
> When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool.
To me, this is "obviously" the wrong approach in most cases. When I'm writing data, I want that write to complete as fast as possible, even at the cost of disk space. That's why I don't save files I'm actively working on in 7zip archives.
But later on, when the system is quiet, I would love for ZFS to go back and figure out which data is duplicated, and use the BRT or whatever to reclaim space. This could be part of a normal scrub operation.
Re: OpenZFS deduplication is good now and you shouldn't use it
#6I wonder why they are having so much trouble getting this working properly with smaller RAM footprints. We have been using commercial storage appliances that have been able to do this for about a decade (at least) now, even on systems with "little" RAM (compared to the amount of disk storage attached). Just store fingerprints in a database and run through that at night and fixup the block pointers...
Fixup block pointers is the one thing ZFS didn't want to do.
Re: OpenZFS deduplication is good now and you shouldn't use it
#7I wonder why they are having so much trouble getting this working properly with smaller RAM footprints. We have been using commercial storage appliances that have been able to do this for about a decade (at least) now, even on systems with "little" RAM (compared to the amount of disk storage attached). Just store fingerprints in a database and run through that at night and fixup the block pointers...
> and fixup the block pointers
That's why. Due to reasons[1], ZFS does not have the capability to rewrite block pointers. It's been a long requested feature[2] as it would also allow for defragmentation.
I've been thinking this could be solved using block pointer indirection, like virtual memory, at the cost of a bit of speed.
But I'm by no means a ZFS developer, so there's surely something I'm missing.
[1]: http://eworldproblems.mbaynton.com/posts/2014/zfs-block-poin...
Re: OpenZFS deduplication is good now and you shouldn't use it
#8If writing performance is critical, why bother with deduplication at writing time? Do deduplication afterwards, concurrently and with lower priority?
Re: OpenZFS deduplication is good now and you shouldn't use it
#9If writing performance is critical, why bother with deduplication at writing time? Do deduplication afterwards, concurrently and with lower priority?
Kinda like log structured merge tree?
Re: OpenZFS deduplication is good now and you shouldn't use it
#10I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…
The author of the new file-based block cloning code had this in mind. A backround process would scan files and identify dupes, delete the dupes and replace them with cloned versions.
There are of course edge cases to consider to avoid data loss, but I imagine it might come soon, either officially or as a third-party tool.