I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.
OpenZFS deduplication is good now and you shouldn't use it
11–20 of 250 posts
Re: OpenZFS deduplication is good now and you shouldn't use it
#12I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…
The ability to alter existing snapshots, even in ways that fully preserve the data, is extremely limited in ZFS. So yes that would be great, but if I was holding my breath for Block Pointer Rewrite I'd be long dead.
Re: OpenZFS deduplication is good now and you shouldn't use it
#13I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.
Internally ZFS is essentially an object store. There was some work which tried to expose it through an object store API. Sadly it seems to not have gone anywhere.
Tried to find the talk but failed, was sure I had seen it on a Delveloper Summit but alas.
Re: OpenZFS deduplication is good now and you shouldn't use it
#14Edit: disregard this, I was wrong and missed the comment deletion window.
Re: OpenZFS deduplication is good now and you shouldn't use it
#15I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools c…
I get the use case, but in most cases (and particularly this one) I'm sure it would be much better to implement that client-side.
You may have seen in the WARC standard that they already do de-duplication based on hashes and use pointers after the first store. So this is exactly a case where FS-level dedup is not all that good.
Re: OpenZFS deduplication is good now and you shouldn't use it
#16Edit: disregard this, I was wrong and missed the comment deletion window.
[deleted]
Re: OpenZFS deduplication is good now and you shouldn't use it
#17Edit: disregard this, I was wrong and missed the comment deletion window.
HN will automatically redirect the submitter to a recent submission instead of allowing a new post... if it had a significant number of comments.
Re: OpenZFS deduplication is good now and you shouldn't use it
#18I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools c…
I get the use case, but in most cases (and particularly this one) I'm sure it would be much better to implement that client-side. You may have seen in the WARC standard that they already do de-duplication based on hashes and use pointers after the first store. So this is exactly a case where FS-level dedup is not all that good.
WARC only does deduping within a single WARC, I'm talking about deduping across millions of WARCs.
Re: OpenZFS deduplication is good now and you shouldn't use it
#19I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…
> I just wish we had "offline" dedupe, or even "lazy" dedupe...
This is the Windows dedupe methodology. I've used it pretty extensively and I'm generally happy with it when the underlying hardware is sufficient. It's very RAM and I/O hungry but you can schedule and throttle the "groveler".
I have had some data eating corruption from bugs in the Windows 2012 R2 timeframe.
Re: OpenZFS deduplication is good now and you shouldn't use it
#20I clicked because of the bait-y title, but ended up reading pretty much the whole post, even though I have no reason to be interested in ZFS. (I skipped most of the stuff about logs...) Everything was explained clearly, I enjoyed the writing style, and the mobile CSS theme was particularly pleasing to my eyes. (It appears to be Pixyll theme with text set to the all-important #000, although I shouldn't derail this discussion with opinions on contrast ratios...)
For less patient readers, note that the concise summary is at the bottom of the post, not the top.