Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

51–60 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#51

Post by UltraSane »

wongarsu wrote: post #50
UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

I haven't tried it myself, but the widely quoted number for old ZFS dedup is that you need 5GB of RAM for every 1TB of disk space. Considering that 1 TB of disk space currently costs about $15 and 5GB of server RAM about $25, you need a 3:1 dedupe ratio just to break even. If your data is a good fit you might get away with 1GB per TB, but if you are out of luck the 5GB might not even be enough. That's why the article…

Why does it need so much RAM? It should only need to store the block hashes which should not need anywhere near that much RAM. Inline dedupe is pretty much standard on high-end storage arrays nowadays.

Re: OpenZFS deduplication is good now and you shouldn't use it

#52

Post by acdha »

UltraSane wrote: post #45
acdha wrote: post #37

Earlier quoted context omitted.

> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.

What software?

Log rotate, cron, or simply having something like Varnish or Apache log to a pipe which is something like bzip2 or zstd. The main question is whether you want to easily access the current stream - e.g. I had uncompressed logs being forwarded to CloudWatch so I had daemons logging to timestamped files with a post-rotate compression command which would run after the last write.

Re: OpenZFS deduplication is good now and you shouldn't use it

#53

Post by UltraSane »

jorvi wrote: post #46
UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

> In my experience 4KB is my preferred block size That makes sense considering Advanced Format harddrives already have a 4K physical sector size, and if you properly low-level format them (to get rid of the ridiculous Windows XP compatibility) they also have 4K logical sector size. I imagine there might be some real performance benefits to having all of those match up.

In the early days of VMware people had a lot of VMs that were converted from physical machines and this causes a nasty alignment issue between the VMDK blocks and the blocks on your storage array. The effect was to always add one block to every read operation, and in the worst case of reading one block would double the load on the storage array. On NetApp this could only be fixed when the VM wasn't running.

Re: OpenZFS deduplication is good now and you shouldn't use it

#54

Post by UltraSane »

acdha wrote: post #52
UltraSane wrote: post #45

Earlier quoted context omitted.

What software?

Log rotate, cron, or simply having something like Varnish or Apache log to a pipe which is something like bzip2 or zstd. The main question is whether you want to easily access the current stream - e.g. I had uncompressed logs being forwarded to CloudWatch so I had daemons logging to timestamped files with a post-rotate compression command which would run after the last write.

That is one wrinkle of using storage based dedupe/compression is you need to avoid doing compression on the client to avoid compressing already compressed data. When a company I worked at first got their Pure array they were using windows file compression heavily and had to disable it as the storage array was now doing it automatically.

Re: OpenZFS deduplication is good now and you shouldn't use it

#55

Post by DannyBee »

Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

You can use any of the offline dupe finders to do this.

Like jdupes or duperemove.

I sent PR's to both the ZFS folks and the duperemove folks to support the syscalls needed.

I actually have to go followup on the ZFS one, it took a while to review and i realized i completely forget to finish it up.

Re: OpenZFS deduplication is good now and you shouldn't use it

#56

Post by floating-io »

wongarsu wrote: post #50
UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

I haven't tried it myself, but the widely quoted number for old ZFS dedup is that you need 5GB of RAM for every 1TB of disk space. Considering that 1 TB of disk space currently costs about $15 and 5GB of server RAM about $25, you need a 3:1 dedupe ratio just to break even. If your data is a good fit you might get away with 1GB per TB, but if you are out of luck the 5GB might not even be enough. That's why the article…

That doesn't account for OpEx, though, such as power...

Re: OpenZFS deduplication is good now and you shouldn't use it

#57

Post by DannyBee »

LeoPanthera wrote: post #40
Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.

I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again.

Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.

Re: OpenZFS deduplication is good now and you shouldn't use it

#58

Post by nikisweeting »

tiffanyh wrote: post #44

OT: does anyone have a good way to dedupe iCloud Photos. Or my Dropbox photos?

- https://github.com/markfasheh/duperemove

- https://codeberg.org/jbruchon/jdupes / https://www.jdupes.com/

- https://github.com/adrianlopezroche/fdupes

- https://github.com/pauldreik/rdfind

Re: OpenZFS deduplication is good now and you shouldn't use it

#59

Post by wmf »

UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

VMs are known to benefit from dedupe so yes, you'll see benefits there. ZFS is a general-purpose filesystem not just an enterprise SAN so many ZFS users aren't running VMs.

Dedupe/compression works really well on syslog

I apologize for the pedantry but dedupe and compression aren't the same thing (although they tend to be bundled in the enterprise storage world). Logs are probably benefiting from compression not dedupe and ZFS had compression all along.

Re: OpenZFS deduplication is good now and you shouldn't use it

#60

Post by acdha »

UltraSane wrote: post #54
acdha wrote: post #52

Earlier quoted context omitted.

Log rotate, cron, or simply having something like Varnish or Apache log to a pipe which is something like bzip2 or zstd. The main question is whether you want to easily access the current stream - e.g. I had uncompressed logs being forwarded to CloudWatch so I had daemons logging to timestamped files with a post-rotate compression command which would run after the last write.

That is one wrinkle of using storage based dedupe/compression is you need to avoid doing compression on the client to avoid compressing already compressed data. When a company I worked at first got their Pure array they were using windows file compression heavily and had to disable it as the storage array was now doing it automatically.

Definitely. We love building abstraction layers but at some point you really need to make decisions across the entire stack.
Post reply on HN