41–50 of 250 posts
I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.
If writing performance is critical, why bother with deduplication at writing time? Do deduplication afterwards, concurrently and with lower priority?
So many flaws. I want to see the author repeat this across 100TB of random data from multiple clients. He/she/whatever will quickly realize why this feature exists. One scenario I am aware of that uses another filesystem in a cloud setup saved 43% of disk space by using dedupe. No, you won't save much on a client system. That isn't what the feature is made for.
My reaction also. Dedupe is a must have for when you are storing hundreds of VMs. you WILL save so much data and inline dedupe will save a lot of write IO.
"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…
> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.
"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…
That makes sense considering Advanced Format harddrives already have a 4K physical sector size, and if you properly low-level format them (to get rid of the ridiculous Windows XP compatibility) they also have 4K logical sector size. I imagine there might be some real performance benefits to having all of those match up.
"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…
You need to be careful and do staggered updates in the VMs or it'll spectacularly explode but it's possible and quite performant for less than mission critical VMs.
If writing performance is critical, why bother with deduplication at writing time? Do deduplication afterwards, concurrently and with lower priority?
That said, NVMe has changed that balance a lot, and you can afford a lot less before you're bottlenecking the drives.
"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…
If your data is a good fit you might get away with 1GB per TB, but if you are out of luck the 5GB might not even be enough. That's why the article speaks of ZFS dedup having a small sweet spot that your data has to hit, and why most people don't bother
Other file systems tend to prefer offline dedupe which has more favorable economics