This array is a bit long-in-the-tooth and only has 192GB of RAM, but a bit over 40TB of net storage, which would be a 200GB dedup table size using the back-of-the-envelope estimate of 5GB/TB. A more precise calculation on my actual data shows that today's data would allow the dedup table to fit in RAM, but if I ever want to actually use most of the 40TB of storage, I'd need more RAM. I've had a ZFS system swap dedup…
Be aware that ZFS performance rapidly drops off north of 80% utilization, when you head into 90%, you will want to buy a bigger array just to escape the pain.
ZFS has gotten significantly better at 80%, but 90% is painful enough that I almost wish it would reserve 10% a bit more explicitly (maybe like the old Unix systems that would prevent non-root users from using the last 5% of the root partition).
All my arrays send me nightly e-mails at 80% so I'm aware of when I hit there, but on a desktop system that's typically not the case.
Why are enterprise SANs so good at dedupe, but filesystems so bad? We use HPE Nimble (yeah, they changed the name recently but I can't be bothered to remember it), and the space savings are insane for the large filesystems we work with. And there is no performance hit.
Some of this is straight up VM storage volumes for ESX virtual disks, some direct LUNs for our file servers. Our gains are upwards of 70%.
Why are enterprise SANs so good at dedupe, but filesystems so bad? We use HPE Nimble (yeah, they changed the name recently but I can't be bothered to remember it), and the space savings are insane for the large filesystems we work with. And there is no performance hit. Some of this is straight up VM storage volumes for ESX virtual disks, some direct LUNs for our file servers. Our gains are upwards of 70%.
Totally naive question: is this better than you get than simply compressing?
It's not 100% clear to me why explicit deduping blocks would give you any significant benefit over a properly chosen compression algorithm.
But that is exactly what the storage array is doing. What is the advantage?
> When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. Linked clones shouldn’t need that. They likely start out with only references to the original blocks, and then replace them when they change. If so, it’s a different concept (as it would mean that any new dupli…
Linked clones aren't used in ESXi, instant clones and they ARE pretty nifty and heavily used in VDI where you need to spin up many thousands of desktop VMs. But they have to keep track of what blocks change and so ever clone has a delta disk. At the end of the day you are just moving around where this bookkeeping happens. And it is best to happen on a enterprise grade array with ultra optimized inline dedupe like a Pure array.
btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.
This is my favourite hack for the Steam Deck. By switching my SD cards and the internal SSD to btrfs, the space savings are unreal (easily halving used space). Every game gets its own prefix which means a crazy amount of file duplication.
Which tool did you end up using for btrfs? I tried out bees https://github.com/Zygo/bees but it is way too slow.
I've read multiple comments on using dedup for VMs here. Wouldn't it be a lot more efficient for this to be implemented by the hypervisor rather than the filesystem?
I'm a former VMware certified admin. How do you envision this to work? All the data written to the VM's virtual disk will cause blocks to change and the storage array is the best place to keep track of that.
> When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. Linked clones shouldn’t need that. They likely start out with only references to the original blocks, and then replace them when they change. If so, it’s a different concept (as it would mean that any new dupli…
Linked clones aren't used in ESXi, instant clones and they ARE pretty nifty and heavily used in VDI where you need to spin up many thousands of desktop VMs. But they have to keep track of what blocks change and so ever clone has a delta disk. At the end of the day you are just moving around where this bookkeeping happens. And it is best to happen on a enterprise grade array with ultra optimized inline dedupe like a P…
I’m not sure that’s true, because the hypervisor can know which blocks are related to begin with? From what I quoted above it seems that the file system instead does a lookup based on the block content to determine if a block is a dupe (I don’t know if it uses a hash, necessitating processing the whole block, or something like an RB tree, which avoids having to read the whole block if it already differs early from candidates). Unless there is a way to explicitly tell the file system that you are copying blocks for that purpose, and that VMware is actually doing that. If not, then leaving it to the file system or even the storage layer should have a definite impact on performance, albeit in exchange for higher space efficiency because a lookup can deduplicate blocks that are identical but not directly related. This would give a space benefit if you do things like installing the same applications across many VMs after the cloning, but assuming that this isn’t commonly done (I think you should clone after establishing all common state like app installations if possible), then my gut feeling is very much that the performance benefit of more semantic-level hypervisor bookkeeping outweighs the space gains from “dumb” block-oriented fs/storage bookkeeping.
> When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. Linked clones shouldn’t need that. They likely start out with only references to the original blocks, and then replace them when they change. If so, it’s a different concept (as it would mean that any new dupli…
Linked clones aren't used in ESXi, instant clones and they ARE pretty nifty and heavily used in VDI where you need to spin up many thousands of desktop VMs. But they have to keep track of what blocks change and so ever clone has a delta disk. At the end of the day you are just moving around where this bookkeeping happens. And it is best to happen on a enterprise grade array with ultra optimized inline dedupe like a P…
> Linked clones aren't used in ESXi
Huh? What do you mean? They absolutely are. I've made extensive use of them in ESXi/vsphere clusters in situations where I'm spinning up and down many temporary VMs.
This is my favourite hack for the Steam Deck. By switching my SD cards and the internal SSD to btrfs, the space savings are unreal (easily halving used space). Every game gets its own prefix which means a crazy amount of file duplication.
Which tool did you end up using for btrfs? I tried out bees https://github.com/Zygo/bees but it is way too slow.
How established was the drive when you set up bees? In my experience you really want to do the initial pass without snapshots existing, but after that it's pretty smooth.
Linked clones aren't used in ESXi, instant clones and they ARE pretty nifty and heavily used in VDI where you need to spin up many thousands of desktop VMs. But they have to keep track of what blocks change and so ever clone has a delta disk. At the end of the day you are just moving around where this bookkeeping happens. And it is best to happen on a enterprise grade array with ultra optimized inline dedupe like a P…
I’m not sure that’s true, because the hypervisor can know which blocks are related to begin with? From what I quoted above it seems that the file system instead does a lookup based on the block content to determine if a block is a dupe (I don’t know if it uses a hash, necessitating processing the whole block, or something like an RB tree, which avoids having to read the whole block if it already differs early from ca…
Your phrasing sounds like you're unaware that filesystems can also do the same kind of cloning that a hypervisor does, where the initial data takes no storage space and only changes get written.
In fact, it's a much more common feature than active deduplication.
VM drives are just files, and it's weird that you imply a filesystem wouldn't know about the semantics of a file getting copied and altered, and would only understand blocks.