Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

71–80 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#71

Post by tw04 »

Joe_Cool wrote: post #47
UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

Even with the rudimentary Dedup features of NTFS on a Windows Hyper-V Server all running the same base image I can overprovision the 512GB partition to almost 2 GB. You need to be careful and do staggered updates in the VMs or it'll spectacularly explode but it's possible and quite performant for less than mission critical VMs.

I think you mean 2TB volume? But yes, this works. But also: if you're doing anything production, I'd strongly recommend doing deduplication on the back-end storage array, not at the NTFS layer. It'll be more performant and almost assuredly have better space savings.

Re: OpenZFS deduplication is good now and you shouldn't use it

#73

Post by UltraSane »

abrookewood wrote: post #68
UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

Couple of comments. Firstly, you are talking about highly redundant information when referencing VM images (e.g. the C drive on all Windows Serer images will be virtually identical), whereas he was using his own laptop contents as an example. Secondly, I think you are conflating two different features: compression & de-duplication. In ZFS you can have compression turned on (almost always worth it) for a pool, but sti…

Fair point. My experience is with enterprise storage arrays and I have always used dedupe/compression at the same time. Dedupe is going to be a lot less useful on single computers.

I consider dedupe/compression to be two different forms of the same thing. compression reduces short range duplication while deduplication reduces long range duplication of data.

Re: OpenZFS deduplication is good now and you shouldn't use it

#74

Post by UltraSane »

tw04 wrote: post #69
wmf wrote: post #59

Earlier quoted context omitted.

VMs are known to benefit from dedupe so yes, you'll see benefits there. ZFS is a general-purpose filesystem not just an enterprise SAN so many ZFS users aren't running VMs. Dedupe/compression works really well on syslog I apologize for the pedantry but dedupe and compression aren't the same thing (although they tend to be bundled in the enterprise storage world). Logs are probably benefiting from compression not dedu…

They are not the same thing, but when you boil it down to the raw math, they aren't identical twins, but they're absolutely fraternal twins. Both are trying to eliminate repeating data, it's just the frame of reference that changes. Compression in this context is operating on a given block or handful of blocks. Deduplication is operating on the entire "volume" of data. "Volume" having a different meaning depending on…

Well put. I like to say compression is just short range dedupe. Hash based dedupe wouldn't be needed if you could just to real-time LZMA on all of the data on a storage array but that just isn't feasible and hash-based dedupe is a very effective compromise.

Re: OpenZFS deduplication is good now and you shouldn't use it

#75

Post by aidenn0 »

UltraSane wrote: post #33
Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

The neat thing about inline dedupe is that if the block hash already exists than the block doesn't have to be written. This can save a LOT of write IO in many situations. There are even extensions where a file copy between to VMs on a dedupe storage array will not actually copy any data but just increment the original blocks reference counter. You will see absurd TB/s write speeds in the OS, it is pretty cool.

This is only a win if the dedupe table fits in RAM; otherwise you pay for it in a LOT of read IO. I have a storage array where dedupe would give me about a 2.2x reduction in disk usage, but there isn't nearly enough RAM for it.

Re: OpenZFS deduplication is good now and you shouldn't use it

#76

Post by UltraSane »

aidenn0 wrote: post #75
UltraSane wrote: post #33

Earlier quoted context omitted.

The neat thing about inline dedupe is that if the block hash already exists than the block doesn't have to be written. This can save a LOT of write IO in many situations. There are even extensions where a file copy between to VMs on a dedupe storage array will not actually copy any data but just increment the original blocks reference counter. You will see absurd TB/s write speeds in the OS, it is pretty cool.

This is only a win if the dedupe table fits in RAM; otherwise you pay for it in a LOT of read IO. I have a storage array where dedupe would give me about a 2.2x reduction in disk usage, but there isn't nearly enough RAM for it.

yes inline dedupe has to fit in RAM. Perhaps enterprise storage arrays have spoiled me.

Re: OpenZFS deduplication is good now and you shouldn't use it

#77

Post by chasil »

UltraSane wrote: post #45
acdha wrote: post #37

Earlier quoted context omitted.

> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.

What software?

Logrotate is the rhel utility, likely present in Fedora, that is easily adapted for custom log handling. I still have rhel5 and I use it there.

CentOS made it famous. I don't know if it has a foothold in the Debian family.

Re: OpenZFS deduplication is good now and you shouldn't use it

#78

Post by nikisweeting »

girishso wrote: post #72

Off topic, any tool to deduplicate files across different external Hard disks? Over the years I made multiple copies of my laptop HDD to different external HDDs, ended up with lots of duplicate copies of files.

How would you want the duplicates resolved? Just reported in some interface or would you want the duplicates deleted off some machines automatically?

There are a few different ways you could solve it but it depends on what final outcome you need.

Re: OpenZFS deduplication is good now and you shouldn't use it

#79

Post by remexre »

UltraSane wrote: post #51
wongarsu wrote: post #50

Earlier quoted context omitted.

I haven't tried it myself, but the widely quoted number for old ZFS dedup is that you need 5GB of RAM for every 1TB of disk space. Considering that 1 TB of disk space currently costs about $15 and 5GB of server RAM about $25, you need a 3:1 dedupe ratio just to break even. If your data is a good fit you might get away with 1GB per TB, but if you are out of luck the 5GB might not even be enough. That's why the article…

Why does it need so much RAM? It should only need to store the block hashes which should not need anywhere near that much RAM. Inline dedupe is pretty much standard on high-end storage arrays nowadays.

    (5GiB / 1TiB) * 4KiB to bits

      ((5 gibibytes) / (1 tebibyte)) × (4 kibibytes) = 160 bits

Re: OpenZFS deduplication is good now and you shouldn't use it

#80

Post by pezezin »

acdha wrote: post #37
UltraSane wrote: post #30

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…

> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.

Yes, that ratio is very small.

I built a very simple, custom syslog solution, a syslog-ng server writing directly to a TimescaleDB hypertable (https://www.timescale.com/) that is then presented as a Grafana dashboard, and I am getting a 30x compression ratio.

Post reply on HN