Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

91–100 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#91

Post by phil21 »

UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

Base VM images would be a rare and specific workload. One of the few cases dedupe makes sense. However you are likely using better strategies like block or filesystem cloning if you are doing VM hosting off a ZFS filesystem. Not doing so would be throwing away one of it's primary differentiators as a filesystem in such an environment.

General purpose fileserving or personal desktop/laptop use generally has very few duplicated blocks and is not worth the overhead. Backups are hit or miss depending on both how the backups are implemented, and if they are encrypted prior to the filesystem level.

Compression is a totally different thing and current ZFS best-practice is to enable it by default for pretty much every workload - the CPU used is barely worth mentioning these days, and the I/O savings can be considerable ignoring any storage space savings. Log storage is going to likely see a lot better than 6:1 savings if you have typical logging, at least in my experience.

Re: OpenZFS deduplication is good now and you shouldn't use it

#92

Post by EasyMark »

UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

I figured he was mostly talking about using dedup on your work (dev machine) computer or family computer at home, not on something like a cloud or streaming server or other back end type operations.

Re: OpenZFS deduplication is good now and you shouldn't use it

#93

Post by m463 »

UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

I would think VMs qualify as a specific workload, since cloning is almost a given.

Re: OpenZFS deduplication is good now and you shouldn't use it

#94

Post by rodarmor »

General-purpose deduplication sounds good in theory but tends not to work out in practice. IPFS uses a rolling hash with variable-sized pieces, in an attempt to deduplicate data rysnc-style. However, in practice, it doesn't actually make a difference, and adds complexity for no reason.

Re: OpenZFS deduplication is good now and you shouldn't use it

#95

Post by bobmcnamara »

UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

> In my experience 4KB is my preferred block size.

This probably has something to do with the VM's filesystem block size. If you have a 4KB filesystem and an 8KB file, the file might be fragmented differently but is still the same 2x4KB blocks just in different places.

Now I wonder if filesystems zero the slack space at the end of the last block in a file in hopes of better host compression. Vs leaving it as past bytes.

Re: OpenZFS deduplication is good now and you shouldn't use it

#96

Post by aidenn0 »

UltraSane wrote: post #76
aidenn0 wrote: post #75

Earlier quoted context omitted.

This is only a win if the dedupe table fits in RAM; otherwise you pay for it in a LOT of read IO. I have a storage array where dedupe would give me about a 2.2x reduction in disk usage, but there isn't nearly enough RAM for it.

yes inline dedupe has to fit in RAM. Perhaps enterprise storage arrays have spoiled me.

This array is a bit long-in-the-tooth and only has 192GB of RAM, but a bit over 40TB of net storage, which would be a 200GB dedup table size using the back-of-the-envelope estimate of 5GB/TB.

A more precise calculation on my actual data shows that today's data would allow the dedup table to fit in RAM, but if I ever want to actually use most of the 40TB of storage, I'd need more RAM. I've had a ZFS system swap dedup to disk before, and the performance dropped to approximately zero; fixing it was a PITA, so I'm not doing that anytime soon.

Re: OpenZFS deduplication is good now and you shouldn't use it

#97

Post by E39M5S62 »

abrookewood wrote: post #88
UltraSane wrote: post #73

Earlier quoted context omitted.

Fair point. My experience is with enterprise storage arrays and I have always used dedupe/compression at the same time. Dedupe is going to be a lot less useful on single computers. I consider dedupe/compression to be two different forms of the same thing. compression reduces short range duplication while deduplication reduces long range duplication of data.

Yeah agreed, very closely related - even more so on ZFS where the compression (AFAIK) is on a block level rather than a file level.

ZFS compression is for sure at the block level - it's fully transparent to the userland tools.

Re: OpenZFS deduplication is good now and you shouldn't use it

#98

Post by aidenn0 »

hhdhdbdb wrote: post #83

Any timing attacks possible on a virtualized system using dedupe? Eg find out what my neighbours have installed. Or if the data before an SSH key is predictable, keep writing that out to disk guessing the next byte or something like that.

I don't think you even need timing attacks if you can read the zpool statistics; you can ask for a histogram of deduped blocks.

Guessing one byte at a time is not possible though because dedupe is block-level in ZFS.

Re: OpenZFS deduplication is good now and you shouldn't use it

#99

Post by E39M5S62 »

chasil wrote: post #77
UltraSane wrote: post #45

Earlier quoted context omitted.

What software?

Logrotate is the rhel utility, likely present in Fedora, that is easily adapted for custom log handling. I still have rhel5 and I use it there. CentOS made it famous. I don't know if it has a foothold in the Debian family.

logrotate is used on Debian and plenty of other distros. It seems pretty widely used, though maybe not as much so now that things log through systemd.

Re: OpenZFS deduplication is good now and you shouldn't use it

#100

Post by edelbitter »

eek2121 wrote: post #28

So many flaws. I want to see the author repeat this across 100TB of random data from multiple clients. He/she/whatever will quickly realize why this feature exists. One scenario I am aware of that uses another filesystem in a cloud setup saved 43% of disk space by using dedupe. No, you won't save much on a client system. That isn't what the feature is made for.

I tried two of the most non-random archives I had and was disappointed just as the author. For mail archives, I got 10%. For entire filesystems, I got.. just as much as with any other COW. Because indeed, I duplicate them only once. Later shared blocks are all over the place.
Post reply on HN