What happened to the issue with ZFS which occurred around half a year go?
I never changed a thing (because it also had some cons) and am believing that as long as a ZFS scrub shows no errors, all is OK. Could I be not seeing a problem?
OpenZFS deduplication is good now and you shouldn't use it
171–180 of 250 posts
Re: OpenZFS deduplication is good now and you shouldn't use it
#172Earlier quoted context omitted.
Why is it a disaster and what would you replace it with? Is the AWS S3 style API an improvement?
One way it's a disaster is that file names (on Linux at least, haven't used Windows in a long time) are byte strings that can contain directory paths from different/multiple file systems. So if you have non-ASCII characters in your paths, encoding/decoding is guesswork, and at worst, differs from path segment to path segment, and there's no metadata attached which encoding to use.
That definitely does not sound like much fun to deal with.
Re: OpenZFS deduplication is good now and you shouldn't use it
#173We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]
I've read multiple comments on using dedup for VMs here. Wouldn't it be a lot more efficient for this to be implemented by the hypervisor rather than the filesystem?
I'm a former VMware certified admin. How do you envision this to work? All the data written to the VM's virtual disk will cause blocks to change and the storage array is the best place to keep track of that.
Re: OpenZFS deduplication is good now and you shouldn't use it
#174Earlier quoted context omitted.
ZFS compression is for sure at the block level - it's fully transparent to the userland tools.
It could be at a file level and still transparent to user land tools, FYI. Depending on what you mean by ‘file level’, I guess.
Windows NTFS has transparent file level compression that works quite well.
Re: OpenZFS deduplication is good now and you shouldn't use it
#175"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads." This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compressi…
Certainly it makes sense to not have deep copies of VM base images, but the deduplication is not the right way to do it in ZFS. Instead, you can clone the base image and before changes it will take almost no space at all. This is thanks to the copy-on-write nature of ZFS. ZFS deduplication instead tries to find existing copies of data that is being written to the volume. For some use cases it could make a lot of sens…
When a new VM is created from a template on a ZFS file system with dedupe enabled what actually happens? Isn't the ref count of every block of the template simply incremented by one? The only time new data will actually be stored is when a block hash a hash that doesn't already exist.
Re: OpenZFS deduplication is good now and you shouldn't use it
#176I clicked because of the bait-y title, but ended up reading pretty much the whole post, even though I have no reason to be interested in ZFS. (I skipped most of the stuff about logs...) Everything was explained clearly, I enjoyed the writing style, and the mobile CSS theme was particularly pleasing to my eyes. (It appears to be Pixyll theme with text set to the all-important #000, although I shouldn't derail this dis…
That being: > As we’ve seen from the last 7000+ words, the overheads are not trivial. Even with all these changes, you still need to have a lot of deduplicated blocks to offset the weight of all the unique entries in your dedup table. [...] what might surprise you is how rare it is to find blocks eligible for deduplication are on most general purpose workloads. > But the real reason you probably don’t want dedup thes…
> offset the weight of all the unique entries in your dedup table
Didn't read the 7000 words... But isn't the dedup table in the form of a bunch of bloom filters so the whole dedup table can be stored with ~1 bit per block?
When you know there is likely a duplicate, you can create a table of blocks where there is a likely duplicate, and find all the duplicates in a single scan later.
That saves having massive amounts of accounting overhead storing any per-block metadata.
Re: OpenZFS deduplication is good now and you shouldn't use it
#177I'd love if dedicated hardware existing in disk controllers for calculating stuff like ECC could be enhanced to expose hashes of blocks to the system. Getting this for free for all your I/O would allow some pretty awesome things.
That is a neat idea. Hard drives could do dedupe from the ECC they calculate for each sector. The main issue with that is that the current ECC is optimal for detecting bit errors but doesn't have the same kind of statistical guarantee of uniqueness that SHA256 or MetroHash has. You need to be VERY confident of the statistical properties of the hash used in dedupe if you are going to increment the ref count of the block hash instead of writing the data to disk.
Re: OpenZFS deduplication is good now and you shouldn't use it
#178What are the use cases where it makes sense to use de-dup? Backup comes to mind. What else?
VMware + All flash storage array + inline dedupe/compression = happy users.
Re: OpenZFS deduplication is good now and you shouldn't use it
#179Any timing attacks possible on a virtualized system using dedupe? Eg find out what my neighbours have installed. Or if the data before an SSH key is predictable, keep writing that out to disk guessing the next byte or something like that.
VMWare ESXi used to dedupe RAM and had to disable this by default because of a security issue it caused that leaded data between VMs.
Re: OpenZFS deduplication is good now and you shouldn't use it
#180Off topic, any tool to deduplicate files across different external Hard disks? Over the years I made multiple copies of my laptop HDD to different external HDDs, ended up with lots of duplicate copies of files.
dupeGuru works pretty well.