Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

191–200 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#191

Post by anyfoo »

UltraSane wrote: post #188
wang_li wrote: post #185

Earlier quoted context omitted.

You do it at the file system layer. Clone the template which creates only metadata referencing the original blocks then you perform copy-on-write as needed.

But that is exactly what the storage array is doing. What is the advantage?

> When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool.

Linked clones shouldn’t need that. They likely start out with only references to the original blocks, and then replace them when they change. If so, it’s a different concept (as it would mean that any new duplicate blocks are not shared), but for the use case of “spin up a hundred identical VMs that only change comparably little” it sounds more efficient performance-wise, with a negligible loss in space efficiency.

Am I certain of this? No, this is just what I quickly pieced together based on some assumptions (albeit reasonable ones). Happy to be told otherwise.

Re: OpenZFS deduplication is good now and you shouldn't use it

#192

Post by p_l »

hinkley wrote: post #35
Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

I get the feeling that a hypothetical ZFS maintainer reading some literature on concurrent mark and sweep would be... inspirational, if not immediately helpful. You should be able to detect duplicates online. Low priority sweeping is something else. But you can at least reduce pause times.

They were aware. The reasons it works the way it works are due to higher priority decisions regarding reliability in face of hardware or software corruption.

That said, you can still do a two-space GC, but it's slow and possibly wasteful.

Re: OpenZFS deduplication is good now and you shouldn't use it

#193

Post by p_l »

perlgeek wrote: post #148
UltraSane wrote: post #41

Earlier quoted context omitted.

Why is it a disaster and what would you replace it with? Is the AWS S3 style API an improvement?

One way it's a disaster is that file names (on Linux at least, haven't used Windows in a long time) are byte strings that can contain directory paths from different/multiple file systems. So if you have non-ASCII characters in your paths, encoding/decoding is guesswork, and at worst, differs from path segment to path segment, and there's no metadata attached which encoding to use.

ZFS actually has settings related to that which originated from providing filesystems for different OSes, where it enforces canonical utf-8 with a specific canonization rule. AFAIK the reason for it existing was cooperation between Solaris, Linux, Windows, and Mac OS X computers all sharing same network filesystem hosted from ZFS.

Re: OpenZFS deduplication is good now and you shouldn't use it

#194

Post by beng-nl »

aidenn0 wrote: post #98
hhdhdbdb wrote: post #83

Any timing attacks possible on a virtualized system using dedupe? Eg find out what my neighbours have installed. Or if the data before an SSH key is predictable, keep writing that out to disk guessing the next byte or something like that.

I don't think you even need timing attacks if you can read the zpool statistics; you can ask for a histogram of deduped blocks. Guessing one byte at a time is not possible though because dedupe is block-level in ZFS.

Gosh, you’re likely right, but what if comparing the blocks (to decide on deduping) is a byte at a time and somehow that can be detected (with a timing channel or a uarch side channel)? Zfs likely compares the hash, but I think KSM doesn’t use hashes but memcmp (or something in that spirit) to avoid collisions. So just maybe… just maybe GP is onto something.. interesting fantasy ;-)

Re: OpenZFS deduplication is good now and you shouldn't use it

#195

Post by mlfreeman »

cmiller1 wrote: post #181

So if a sweet spot exists where dedup is widely beneficial then: Is there an easy way to analyze your dataset to find if you're in this sweet spot? If so, is anyone working on some kind of automated partial dedup system where only portions of the filesystem are dedupped based on analysis of how beneficial it would be?

Are there any tools that can run (even across network on another box) to analyze possible duplication at various block sizes?

I am NOT interested in finding duplicate files, but duplicate slices within all my files overall.

I can easily throw together code myself to find duplicate files.

EDIT: I guess I’m looking for a ZFS/BTRFS/other dedupe preview tool that would say “you might save this much if you used this dedupe process.”

Re: OpenZFS deduplication is good now and you shouldn't use it

#196

Post by iwontberude »

aniviacat wrote: post #164
simonjgreen wrote: post #137

We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]

I've read multiple comments on using dedup for VMs here. Wouldn't it be a lot more efficient for this to be implemented by the hypervisor rather than the filesystem?

COW is significantly slower and has nesting limits when compared to these deduped clones. Great question!

Re: OpenZFS deduplication is good now and you shouldn't use it

#197

Post by bastloing »

Joel_Mckay wrote: post #31
bastloing wrote: post #27

Forget dedupe just use zfs compression, a lot more bang for your buck

Unless your data-set is highly compressed media files. In general, even during rsync operations one often turns off compression on large video files, as the compression operation has low or negative impact on storage/transfers while eating ram and cpu power. De-duplication is good for Virtual Machine OS images, as the majority of the storage cost is a replicated backing image. =3

Compression is still king. Check out HP's Nimble storage arrays. Way quicker to do compression, fewer iops, and less overhead. Even when it misses, like video files, it's still a winner.

Re: OpenZFS deduplication is good now and you shouldn't use it

#198

Post by Joe_Cool »

tw04 wrote: post #71
Joe_Cool wrote: post #47

Earlier quoted context omitted.

Even with the rudimentary Dedup features of NTFS on a Windows Hyper-V Server all running the same base image I can overprovision the 512GB partition to almost 2 GB. You need to be careful and do staggered updates in the VMs or it'll spectacularly explode but it's possible and quite performant for less than mission critical VMs.

I think you mean 2TB volume? But yes, this works. But also: if you're doing anything production, I'd strongly recommend doing deduplication on the back-end storage array, not at the NTFS layer. It'll be more performant and almost assuredly have better space savings.

For sure it's not for production. At least not for stuff that's critical. MS also doesn't recommend using it for live VHDX.

The partition/NTFS volume is 512GB. It currently stores 1.3 TB of "dedupped" data and has about 200GB free. Dedup runs asynchronously in the background and as a job during off hours.

It's a typo, yes. Thanks.

Re: OpenZFS deduplication is good now and you shouldn't use it

#199

Post by volkadav »

burnt-resistor wrote: post #157

Already don't use ZoL because of their history of arms shrug-level support coupled with a lack of QA. ZoL != Solaris ZFS. It is mostly an aspirational cargo cult. Only a few fses like XFS and Ext4 have meaningful real-world, enterprise deployment hours. Technically, btrfs has significant (web ops instead of IT ops) deployment exposure due to its use on 10M boxes at Meta. Many non-mainstream fses also aren't assured t…

I sympathize with your concerns for stability and testing, but I think that you might reconsider things in open-source ZFS land. OpenZFS/ZoL have been merged since the 2.0 release several years back, and some very large (e.g. Netflix) environments use FreeBSD which in turn uses OpenZFS, as well as being in use by the various Illumos derivatives and such. It is true that there has been some feature divergence between Oracle ZFS and OpenZFS since the fork, but as I recall that was more "nice to haves" like fs-native encryption than essential reliability fixes, fwiw.

Re: OpenZFS deduplication is good now and you shouldn't use it

#200

Post by forrestthewoods »

fragmede wrote: post #90
forrestthewoods wrote: post #63

My dream Git successor would use either dedupe or a simple cache plus copy-on-write so that repos can commit toolchains and dependencies and users wouldn’t need to worry about disk drive bloat. Maybe someday…

It does dedup using Sha-1 on entire files. you might try git-lfs for your usecase though.

Git LFS is a really really bad gross hack. It’s awful.

https://www.forrestthewoods.com/blog/dependencies-belong-in-...

Post reply on HN