Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

161–170 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#161

Post by nisten »

spockz wrote: post #109
UltraSane wrote: post #39

Earlier quoted context omitted.

Generally the smaller the dedupe block the better as you are far more likely to find a matching block. But larger blocks will reduce the number of hashes you have to store. In my experience 4KB is the sweet spot to maximize how much data you save.

So in this case I think it would make sense to have a separate pool where you store large files like media so you can save on the dedup for them. Is there an inherent performance loss of using 64kB blocks on FS level when using storage devices that are 4kB under the hood?

Hmmm you might be able to do both no? Like the dedube is gonna run at the filesystem level but your memory security & ownership stuff is gonna run more efficiently. I am not sure.

Re: OpenZFS deduplication is good now and you shouldn't use it

#162

Post by rwarfield »

jamesbfb wrote: post #159
simonjgreen wrote: post #137

We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]

Can relate. I’ve recently taken ownership of a new work laptop with Ubuntu (with “experimental” zfs) and using dedupe on my nix store has been an absolute blessing!

Isn't it better to use `nix store optimise` for dedup of the nix store? The nix command has more knowledge of the structure of the nix store so should be able to do a better job with fewer resources. Also the store is immutable so you don't actually need reflinks - hard links are enough.

Re: OpenZFS deduplication is good now and you shouldn't use it

#163

Post by amarshall »

jamesbfb wrote: post #159
simonjgreen wrote: post #137

We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]

Can relate. I’ve recently taken ownership of a new work laptop with Ubuntu (with “experimental” zfs) and using dedupe on my nix store has been an absolute blessing!

Nix already has some builtin deduplication, see `man nix-store-optimise`. Nix’s own hardlinking optimization reduces disk usage of the store (for me) by 30–40%.

Re: OpenZFS deduplication is good now and you shouldn't use it

#164

Post by aniviacat »

simonjgreen wrote: post #137

We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]

I've read multiple comments on using dedup for VMs here. Wouldn't it be a lot more efficient for this to be implemented by the hypervisor rather than the filesystem?

Re: OpenZFS deduplication is good now and you shouldn't use it

#165

Post by myself248 »

xmodem wrote: post #146

We have a read-heavy zpool with some data that's used as part of our build process, on which we see a roughly 8x savings with dedup - and because of this ZFS dedup makes it economically viable for us to store the pool on NVMe rather than spinning rust.

And being read-heavy, suboptimal performance at write time is an infrequent pain, I guess?

Re: OpenZFS deduplication is good now and you shouldn't use it

#166

Post by xmodem »

myself248 wrote: post #165
xmodem wrote: post #146

We have a read-heavy zpool with some data that's used as part of our build process, on which we see a roughly 8x savings with dedup - and because of this ZFS dedup makes it economically viable for us to store the pool on NVMe rather than spinning rust.

And being read-heavy, suboptimal performance at write time is an infrequent pain, I guess?

Not even that - the data being written is coming straight from the network, and the pool has no issues keeping up.

Re: OpenZFS deduplication is good now and you shouldn't use it

#167

Post by Pet_Ant »

cryptonector wrote: post #85
Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

Lazy/off-line dedup requires block pointer rewrite, but ZFS _cannot_ and will not ever get true BP rewrite because ZFS is not truly a CAS system. The problem is that physical locations are hashed into the Merkle hash tree, and that makes moving physical locations prohibitively expensive as you have to rewrite all the interior nodes on the way to the nodes you want to rewrite. A better design would have been to split…

> CAS system

It looks like it means: https://en.wikipedia.org/wiki/Content-addressable_storage

Re: OpenZFS deduplication is good now and you shouldn't use it

#168

Post by Wowfunhappy »

Dylan16807 wrote: post #86
Wowfunhappy wrote: post #70

Earlier quoted context omitted.

But then you have to be careful not to remove the one which happens to be the "original" or the hardlinks will break, right?

No, pointing to an original is how soft links work. Hard links are all equivalent. A file has any number of hard links, and at least in theory you can't distinguish between them. The risk with hardlinks is that you might alter the file. Reflinks remove that risk, and also perform very well.

Thank you, I was unaware of this.

However, the fact that editing one copy edits all of them still makes this a non-solution for me at least. I'd also strongly prefer deduping at the block level vs file level.

Re: OpenZFS deduplication is good now and you shouldn't use it

#169

Post by gnu8 »

barrkel wrote: post #138
aidenn0 wrote: post #96

Earlier quoted context omitted.

This array is a bit long-in-the-tooth and only has 192GB of RAM, but a bit over 40TB of net storage, which would be a 200GB dedup table size using the back-of-the-envelope estimate of 5GB/TB. A more precise calculation on my actual data shows that today's data would allow the dedup table to fit in RAM, but if I ever want to actually use most of the 40TB of storage, I'd need more RAM. I've had a ZFS system swap dedup…

Be aware that ZFS performance rapidly drops off north of 80% utilization, when you head into 90%, you will want to buy a bigger array just to escape the pain.

I think that is well known amongst storage experts, though maybe not everyone who might be interested in using ZFS for storage in a professional or personal application. What I’m curious about is how ZFS’s full-disk performance (what is the best term for this?) compares to btrfs, WAFL, and so on. Is ZFS abnormally sensitive to this condition, or is it a normal property?

In any case it doesn’t stick out to me as a problem that needs to be fixed. You can’t fill a propane tank to 100% either.

Re: OpenZFS deduplication is good now and you shouldn't use it

#170

Post by BodyCulture »

eadmund wrote: post #151
BodyCulture wrote: post #149

I wanted to use ZFS badly, but of course all data must be encrypted. It was surprising to see how usage gets much more complicated than expected and so many people just don’t encrypt their data because things get wild then. Look, even Proxmox, which I totally expected to support encryption with default installation (it has „Enterprise“ on the website) does loose important features when trying to use with encryption.…

The best way to encrypt ZFS is to run unecrypted ZFS atop encrypted volumes (e.g. LUKS volumes). ZFS ‘encryption’ leaves too much in plaintext for my comfort.

In the Proxmox forum some people tried this method and do not report big success. Can not recommend for production.

Still the same picture, encryption seems to be not a first class citizen in ZFS land.

Post reply on HN