Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

231–240 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#231

Post by Maakuth »

UltraSane wrote: post #175
Maakuth wrote: post #124

Earlier quoted context omitted.

Certainly it makes sense to not have deep copies of VM base images, but the deduplication is not the right way to do it in ZFS. Instead, you can clone the base image and before changes it will take almost no space at all. This is thanks to the copy-on-write nature of ZFS. ZFS deduplication instead tries to find existing copies of data that is being written to the volume. For some use cases it could make a lot of sens…

When a new VM is created from a template on a ZFS file system with dedupe enabled what actually happens? Isn't the ref count of every block of the template simply incremented by one? The only time new data will actually be stored is when a block hash a hash that doesn't already exist.

That's right, though deduplication feature is not the way to do it. The VM template would be a zvol, which is a block device backed by the lower levels of ZFS, and it would be cloned to a new zvol for each VM. Alternatively, if image files were used, the image file could be a reflinked copy. In both cases, new data would be stored only when changes accumulate.

Compare this to the deduplication approach: the filesystem would need to keep tabs on the data that's already on disk, identify the case where the same data is being written and then make that a reference to the existing data instead. Very inefficient if on application level you already know that it is just a copy being made.

In both of these cases, you could say that the data ends up being deduplicated. But the second approach is what the deduplication feature does. The first one is "just" copy-on-write.

Re: OpenZFS deduplication is good now and you shouldn't use it

#233

Post by emptiestplace »

shiroiushi wrote: post #230
emptiestplace wrote: post #229

Earlier quoted context omitted.

Ok, understood. The part I'm confused about is the focus on NVMe devices - do you also have a bunch of SATA/SAS SSDs, or even conventional disks for your media? If not, I'd definitely start there. Maybe something like six spinners in RAIDZ2, this would allow you to lose up to two drives without any data loss. If NVMe is your only option, I'd try to find a couple used 1.92TB enterprise class drives on ebay, and go ahe…

>The part I'm confused about is the focus on NVMe devices - do you also have a bunch of SATA/SAS SSDs I do have 4 regular SATA spinning disks (enterprise-class), for bulk data storage, in a RAIDZ1 array. I know it's not as safe as RAIDZ2, but I thought it'd be safe enough with only 4 disks, and I want to keep power usage down if possible. I'm using (right now) a single 512GB NVMe drive for both booting and app storag…

Yeah, RAIDZ1 is a reasonable trade-off for the four disks.

Move your NVMe to the other slot, I bet you can't tell a difference without synthetic benchmarks.

Re: OpenZFS deduplication is good now and you shouldn't use it

#234

Post by spockz »

magicalhippo wrote: post #153
spockz wrote: post #108

Earlier quoted context omitted.

How would this work if I have snapshots? Wouldn’t then the version of the file I just replaced still be in use there? But maybe I also need to store the copy again if I make another snapshot because the “original “ file isn’t part of the snapshot? So now I’m effectively storing more not less?

AFAIK, yes. Blocks are reference counted, so if the duplicate file is in a snapshot then the blocks would be referenced by the snapshot and hence not be eligible for deallocation. Only once the reference count falls to zero would the block be freed. This is par for the course with ZFS though. If you delete a non-duplicated file you don't get the space back until any snapshots referencing the file are deleted.

Yes that snapshots incur a cost I know. But I’m wondering whether now the action of deduplicating actually created an extra copy instead of saving’one.

Re: OpenZFS deduplication is good now and you shouldn't use it

#235

Post by jamesbfb »

amarshall wrote: post #163
jamesbfb wrote: post #159

Earlier quoted context omitted.

Can relate. I’ve recently taken ownership of a new work laptop with Ubuntu (with “experimental” zfs) and using dedupe on my nix store has been an absolute blessing!

Nix already has some builtin deduplication, see `man nix-store-optimise`. Nix’s own hardlinking optimization reduces disk usage of the store (for me) by 30–40%.

Update. Turns out PyCharm does not play nice with a plethora of symlinks. :(

Re: OpenZFS deduplication is good now and you shouldn't use it

#236

Post by kccqzy »

Dylan16807 wrote: post #209
kccqzy wrote: post #205

Earlier quoted context omitted.

Which tool did you end up using for btrfs? I tried out bees https://github.com/Zygo/bees but it is way too slow.

How established was the drive when you set up bees? In my experience you really want to do the initial pass without snapshots existing, but after that it's pretty smooth.

Quite established. Although having read about snapshots causing slowness, I actually deleted all my snapshots and didn't notice any improvement.

Re: OpenZFS deduplication is good now and you shouldn't use it

#237

Post by amarshall »

jamesbfb wrote: post #235
amarshall wrote: post #163

Earlier quoted context omitted.

Nix already has some builtin deduplication, see `man nix-store-optimise`. Nix’s own hardlinking optimization reduces disk usage of the store (for me) by 30–40%.

Update. Turns out PyCharm does not play nice with a plethora of symlinks. :(

Nix optimise does not use symlinks, it uses hardlinks.

Re: OpenZFS deduplication is good now and you shouldn't use it

#238

Post by UltraSane »

PittleyDunkin wrote: post #206
UltraSane wrote: post #173

Earlier quoted context omitted.

I'm a former VMware certified admin. How do you envision this to work? All the data written to the VM's virtual disk will cause blocks to change and the storage array is the best place to keep track of that.

> VMware certified admin Not to be rude, but does this have any meaning?

I understand how VMware ESXi works better than most people.

Re: OpenZFS deduplication is good now and you shouldn't use it

#239

Post by Filligree »

amarshall wrote: post #219
Filligree wrote: post #190

Earlier quoted context omitted.

It is, yeah, though you have to turn it on. I'm not actually sure why it's off by default.

It’s off by default as it can make builds slower (regardless of platform)—you should test this if you care. There also are (or were) some bugs on macOS that would cause corruption.

That seems like the wrong default. Most people do very little building on their desktops; they get all their software from the cache.

Re: OpenZFS deduplication is good now and you shouldn't use it

#240

Post by janc_ »

spockz wrote: post #105
UltraSane wrote: post #82

Earlier quoted context omitted.

What about rack space?

Not just rack space. At a certain amount of disks you also need to get a separate server (chassis + main board + cpu + ram) to host the disks. Maybe you need that for performance reasons any way. But saving disk space and only paying for it with some ram sounds cost effective.

That works out only as long as you don’t have to replace the whole machine (motherboard & possibly CPU/CPUs) to be able to add more RAM… So essentially the same problem as with disks.

In the end it all comes down to: there are a whole lot of trade-offs that you have to take into account, and which ceilings you hit first depends entirely on everyone’s specific situation.

Post reply on HN