Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

111–120 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#111

Post by edelbitter »

DannyBee wrote: post #57
LeoPanthera wrote: post #40

Earlier quoted context omitted.

btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.

I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again. Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.

Knowing what you had to know to write that, would you dare using it?

Compression, encryption and streaming sparse files together are impressive already. But now we get a new BRT entry appearing out of nowhere, dedup index pruning one that was there a moment ago, all while correctly handling arbitrary errors in whatever simultaneous deduped writes, O_DIRECT writes, FALLOC_FL_PUNCH_HOLE and reads were waiting for the same range? Sounds like adding six new places to hold the wrong lock to me.

Re: OpenZFS deduplication is good now and you shouldn't use it

#113

Post by rattt »

DannyBee wrote: post #57
LeoPanthera wrote: post #40

Earlier quoted context omitted.

btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.

I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again. Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.

Shouldn't jdupes like tools already work now that ZFS has reflink copy support?

Re: OpenZFS deduplication is good now and you shouldn't use it

#114

Post by DannyBee »

AndrewDavis wrote: post #101
tiagod wrote: post #67

Earlier quoted context omitted.

I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine! https://github.com/pauldreik/rdfind

So this is great, if you're just looking to deduplicate read only files. Less so if you intend to write to them. Write to one and they're both updated. Anyway. Offline/lazy dedup (not in the zfs dedup sense) is something that could be done in userspace, at the file level on any filesystem that supports reflinks. When a tool like rdfind finds a duplicate, instead of replacing with a hardlink, create a copy of the file…

copy_file_range already works on zfs, but it doesn't guarantee anything interesting.

Basically all dupe tools that are modern use fideduprange, which is meant to tell the FS which things should be sharing data, and let it take care of the rest. (BTRFS, bcachefs, etc support this ioctl, and zfs will soon too)

Unlike copy_file_range, it is meant for exactly this use case, and will tell you how many bytes were dedup'd, etc.

Re: OpenZFS deduplication is good now and you shouldn't use it

#115

Post by DannyBee »

rattt wrote: post #113
DannyBee wrote: post #57

Earlier quoted context omitted.

I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again. Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.

Shouldn't jdupes like tools already work now that ZFS has reflink copy support?

No, because none of these tools use copy_file_range. Because copy_file_range doesn't guarantee deduplication or anything. It is meant to copy data. So you could just end up copying data, when you aren't even trying to copy anything at all.

All modern tools use FIDEDUPRANGE, which is an ioctl meant for explicitly this use case - telling the FS that two files have bytes that should be shared.

Under the covers, the FS does block cloning or whatever to make it happen.

Nothing is copied.

ZFS does support FICLONERANGE, which is the same as FIDEDUPRANGE but it does not verify the contents are the same prior to cloning.

Both are atomic WRT to concurrent writes, but for FIDEDUPRANGE that means the compare is part of the atomicness. So you don't have to do any locking.

If you used FICLONERANGE, you'd need to lock the two file ranges, verify, clone, unlock

FIDEDUPRANGE does this for you.

So it is possible, with no changes to ZFS, to modify dedup tools to work on ZFS by changing them to use FICLONERANGE + locking if FIDEDUPRANGE does not exist.

Re: OpenZFS deduplication is good now and you shouldn't use it

#116

Post by DannyBee »

edelbitter wrote: post #111
DannyBee wrote: post #57

Earlier quoted context omitted.

I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again. Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.

Knowing what you had to know to write that, would you dare using it? Compression, encryption and streaming sparse files together are impressive already. But now we get a new BRT entry appearing out of nowhere, dedup index pruning one that was there a moment ago, all while correctly handling arbitrary errors in whatever simultaneous deduped writes, O_DIRECT writes, FALLOC_FL_PUNCH_HOLE and reads were waiting for the s…

"Knowing what you had to know to write that, would you dare using it?"

It's no worse than anything else related to block cloning :)

ZFS already supports FICLONERANGE, the thing FIDEDUPRANGE changes is that the compare is part of the atomic guarantee.

So in fact, i'd argue it's actually better than what is there now - yes, the hardest part is the locking, but the locking is handled by the dedup range call getting the right locks upfront, and passing them along, so nothing else is grabbing the wrong locks. It actually has to because of the requirements to implement the ioctl properly. We have to be able to read both ranges, compare them, and clone them, all as an atomic operation wrt to concurrent writes. So instead of random things grabbing random locks, we pass the right locks around and everything verifies the locks.

This means fideduprange is not as fast as it maybe could be, but it does not run into the "oops we forgot the right kind of lock" issue. At worst, it would deadlock, because it's holding exclusive locks on all that it could need before it starts to do anything in order to guarantee both the compare and the clone are atomic. So something trying to grab a lock forever under it will just deadlock.

This seemed the safest course of implementation.

ficlonerange is only atomic in the cloning, which means it does not have to read anything first, it can just do blind block cloning. So it actually has a more complex (but theoretically faster) lock structure because of the relaxed constraints.

Re: OpenZFS deduplication is good now and you shouldn't use it

#117

Post by watersb »

I've used ZFS dedupe for a personal archive since dedupe was first introduced.

Currently, it seems to be reducing on-disk footprint by a factor of 3.

When I first started this project, 2TB hard drives were the largest available.

My current setup uses slow 2.5-inch hard drives; I attempt to improve things somewhat via NVMe-based Optane drives for cache.

Every few years, I try to do a better job of things but at this point, the best improvement would be radical simplification.

ZFS has served very well in terms of reliability. I haven't lost data, and I've been able to catch lots of episodes of almost losing data. Or writing the wrong data.

Not entirely sure how I'd replace it, if I want something that can spot bit rot and correct it. ZFS scrub.

Re: OpenZFS deduplication is good now and you shouldn't use it

#118

Post by roygbiv2 »

watersb wrote: post #117

I've used ZFS dedupe for a personal archive since dedupe was first introduced. Currently, it seems to be reducing on-disk footprint by a factor of 3. When I first started this project, 2TB hard drives were the largest available. My current setup uses slow 2.5-inch hard drives; I attempt to improve things somewhat via NVMe-based Optane drives for cache. Every few years, I try to do a better job of things but at this p…

Do you have data that is very obviously dedupeable? Or just a mix of things? A factor of three is not to be sniffed at.

Re: OpenZFS deduplication is good now and you shouldn't use it

#119

Post by DannyBee »

DannyBee wrote: post #57
LeoPanthera wrote: post #40

Earlier quoted context omitted.

btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.

I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again. Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.

Note - anyone bored enough could already make any of these tools work by using FICLONERANGE (which ZFS already supports), but you'd have to do locking - lock, compare file ranges, clone, unlock.

Because FIDEDUPRANGE has the compare as part of the atomic guarantee, you don't need to lock in userspace around using it, and so no dedup utility bothers to do FICLONERANGE + locking. Also, ZFS is the only FS that implements FICLONERANGE but not FIDEDUPRANGE :)

Re: OpenZFS deduplication is good now and you shouldn't use it

#120

Post by emptiestplace »

kderbe wrote: post #20

I clicked because of the bait-y title, but ended up reading pretty much the whole post, even though I have no reason to be interested in ZFS. (I skipped most of the stuff about logs...) Everything was explained clearly, I enjoyed the writing style, and the mobile CSS theme was particularly pleasing to my eyes. (It appears to be Pixyll theme with text set to the all-important #000, although I shouldn't derail this dis…

It scrolls horizontally :(
Post reply on HN