Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

211–220 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#211

Post by hhdhdbdb »

beng-nl wrote: post #194
aidenn0 wrote: post #98

Earlier quoted context omitted.

I don't think you even need timing attacks if you can read the zpool statistics; you can ask for a histogram of deduped blocks. Guessing one byte at a time is not possible though because dedupe is block-level in ZFS.

Gosh, you’re likely right, but what if comparing the blocks (to decide on deduping) is a byte at a time and somehow that can be detected (with a timing channel or a uarch side channel)? Zfs likely compares the hash, but I think KSM doesn’t use hashes but memcmp (or something in that spirit) to avoid collisions. So just maybe… just maybe GP is onto something.. interesting fantasy ;-)

Thanks for putting meat on the (speculitive) bone I threw out! Very interesting.

Re: OpenZFS deduplication is good now and you shouldn't use it

#212

Post by Dylan16807 »

UltraSane wrote: post #174
lazide wrote: post #102

Earlier quoted context omitted.

It could be at a file level and still transparent to user land tools, FYI. Depending on what you mean by ‘file level’, I guess.

Windows NTFS has transparent file level compression that works quite well.

I don't know how much I agree with that.

The old kind of NTFS compression from 1993 is completely transparent, but it uses a weak algorithm and processes each 64KB of file completely independently. It also fragments files to hell and back.

The new kind from Windows 10 has a better algorithm and can have up to 2MB of context, which is quite reasonable. But it's not transparent to writes, only to reads. You have to manually apply it and if anything writes to the file it decompresses.

I've gotten okay use out of both in certain directories, with the latter being better despite the annoyances, but I think they both have a lot of missed potential compared to how ZFS and BTRFS handle compression.

Re: OpenZFS deduplication is good now and you shouldn't use it

#213

Post by Wowfunhappy »

DannyBee wrote: post #57
LeoPanthera wrote: post #40

Earlier quoted context omitted.

btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.

I sent a PR to add support for the necessary syscall (FIDUPERANGE) to zfs that i just have to clean up again. Once that is in, any of the existing dupe finding tools that use it (IE jdupes, duperemove) will just work on ZFS.

Oh cool! Does this work on the block level or only the file level?

Re: OpenZFS deduplication is good now and you shouldn't use it

#214

Post by anyfoo »

Dylan16807 wrote: post #210
anyfoo wrote: post #207

Earlier quoted context omitted.

I’m not sure that’s true, because the hypervisor can know which blocks are related to begin with? From what I quoted above it seems that the file system instead does a lookup based on the block content to determine if a block is a dupe (I don’t know if it uses a hash, necessitating processing the whole block, or something like an RB tree, which avoids having to read the whole block if it already differs early from ca…

Your phrasing sounds like you're unaware that filesystems can also do the same kind of cloning that a hypervisor does, where the initial data takes no storage space and only changes get written. In fact, it's a much more common feature than active deduplication. VM drives are just files, and it's weird that you imply a filesystem wouldn't know about the semantics of a file getting copied and altered, and would only u…

Uh, thanks for the personal attack? I am aware that cloning exists, and I very explicitly allowed for the use of such a mechanism to change the conclusion in both of my comments. My trouble was that I wasn't sure how much filesystem-cloning is actually in use in relevant contexts. Does POSIX have some sort of "copyfile()" system call nowadays? Last I knew (outdated, I'm sure), the cp command for example seemed to just read() blocks into a buffer and write() them out again. I'm not sure how the filesystem layer would detect this as a clone without a lookup. I was quoting and basing my assumptions on the article:

> The downside is that every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool.

Which, if universally true, is very much different from what a hypervisor could do instead, and I've detailed the potential differences. But if a hypervisor does use some sort of clone system call instead, that can indeed shift the same approach into the fs layer, and my genuine question is whether it does.

Re: OpenZFS deduplication is good now and you shouldn't use it

#215

Post by Dylan16807 »

anyfoo wrote: post #214
Dylan16807 wrote: post #210

Earlier quoted context omitted.

Your phrasing sounds like you're unaware that filesystems can also do the same kind of cloning that a hypervisor does, where the initial data takes no storage space and only changes get written. In fact, it's a much more common feature than active deduplication. VM drives are just files, and it's weird that you imply a filesystem wouldn't know about the semantics of a file getting copied and altered, and would only u…

Uh, thanks for the personal attack? I am aware that cloning exists, and I very explicitly allowed for the use of such a mechanism to change the conclusion in both of my comments. My trouble was that I wasn't sure how much filesystem-cloning is actually in use in relevant contexts. Does POSIX have some sort of "copyfile()" system call nowadays? Last I knew (outdated, I'm sure), the cp command for example seemed to jus…

I said "your phrasing sounds like" specifically to make it not personal. Clearly some information was missing but I wasn't sure exactly what. I'll try to phrase that better in the future.

It sounds like the information you need is that cp has a flag to make cloning happen. I think it even became default behavior recently.

Also that the article quote is strictly talking about dedup. That downside does not generalize to the clone/reflink features. They use a much more lightweight method.

This is one of the relevant syscalls: https://man7.org/linux/man-pages/man2/copy_file_range.2.html

Re: OpenZFS deduplication is good now and you shouldn't use it

#216

Post by jamesbfb »

amarshall wrote: post #163
jamesbfb wrote: post #159

Earlier quoted context omitted.

Can relate. I’ve recently taken ownership of a new work laptop with Ubuntu (with “experimental” zfs) and using dedupe on my nix store has been an absolute blessing!

Nix already has some builtin deduplication, see `man nix-store-optimise`. Nix’s own hardlinking optimization reduces disk usage of the store (for me) by 30–40%.

Well, TIL. Being relatively new to nix, you’ve let me down another rabbit hole :)

Re: OpenZFS deduplication is good now and you shouldn't use it

#217

Post by anyfoo »

Dylan16807 wrote: post #215
anyfoo wrote: post #214

Earlier quoted context omitted.

Uh, thanks for the personal attack? I am aware that cloning exists, and I very explicitly allowed for the use of such a mechanism to change the conclusion in both of my comments. My trouble was that I wasn't sure how much filesystem-cloning is actually in use in relevant contexts. Does POSIX have some sort of "copyfile()" system call nowadays? Last I knew (outdated, I'm sure), the cp command for example seemed to jus…

I said "your phrasing sounds like" specifically to make it not personal. Clearly some information was missing but I wasn't sure exactly what. I'll try to phrase that better in the future. It sounds like the information you need is that cp has a flag to make cloning happen. I think it even became default behavior recently. Also that the article quote is strictly talking about dedup. That downside does not generalize t…

Thanks, that was educational.

Re: OpenZFS deduplication is good now and you shouldn't use it

#218

Post by fragmede »

forrestthewoods wrote: post #200
fragmede wrote: post #90

Earlier quoted context omitted.

It does dedup using Sha-1 on entire files. you might try git-lfs for your usecase though.

Git LFS is a really really bad gross hack. It’s awful. https://www.forrestthewoods.com/blog/dependencies-belong-in-...

It's quite functional and usable, now. so I'd agree with hack, just not the rest of your adjectives.

That was a good read! I've been thinking a lot about what comes after git too. One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo.

Re: OpenZFS deduplication is good now and you shouldn't use it

#219

Post by amarshall »

Filligree wrote: post #190
rwarfield wrote: post #162

Earlier quoted context omitted.

Isn't it better to use `nix store optimise` for dedup of the nix store? The nix command has more knowledge of the structure of the nix store so should be able to do a better job with fewer resources. Also the store is immutable so you don't actually need reflinks - hard links are enough.

It is, yeah, though you have to turn it on. I'm not actually sure why it's off by default.

It’s off by default as it can make builds slower (regardless of platform)—you should test this if you care. There also are (or were) some bugs on macOS that would cause corruption.

Re: OpenZFS deduplication is good now and you shouldn't use it

#220

Post by watersb »

roygbiv2 wrote: post #118
watersb wrote: post #117

I've used ZFS dedupe for a personal archive since dedupe was first introduced. Currently, it seems to be reducing on-disk footprint by a factor of 3. When I first started this project, 2TB hard drives were the largest available. My current setup uses slow 2.5-inch hard drives; I attempt to improve things somewhat via NVMe-based Optane drives for cache. Every few years, I try to do a better job of things but at this p…

Do you have data that is very obviously dedupeable? Or just a mix of things? A factor of three is not to be sniffed at.

This archive was created as a dumping ground for all of my various computers, so a complete file by file dump of their system drives.

While that obviously leads to duplicate data from files installed by operating systems, there's a lot of duplicate media libraries. (File-oriented dedupe might not be very effective for old iTunes collections, as iTunes stores metadata like how many times a song has been played in the same file as the audio data. So the hash value of a song will change every time it's played; it looks like a different file. ZFS block-level dedupe might still work ok here because nearly all of the blocks that comprise the song data will be identical.)

Anyway. It's a huge pile of stuff, a holding area of data that should really be sorted into something small and rational.

The application leads to a big payoff for deduplication.

Post reply on HN