Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

151–160 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#151

Post by eadmund »

BodyCulture wrote: post #149

I wanted to use ZFS badly, but of course all data must be encrypted. It was surprising to see how usage gets much more complicated than expected and so many people just don’t encrypt their data because things get wild then. Look, even Proxmox, which I totally expected to support encryption with default installation (it has „Enterprise“ on the website) does loose important features when trying to use with encryption.…

The best way to encrypt ZFS is to run unecrypted ZFS atop encrypted volumes (e.g. LUKS volumes). ZFS ‘encryption’ leaves too much in plaintext for my comfort.

Re: OpenZFS deduplication is good now and you shouldn't use it

#152

Post by armada651 »

XorNot wrote: post #145
phil21 wrote: post #91

Earlier quoted context omitted.

Base VM images would be a rare and specific workload. One of the few cases dedupe makes sense. However you are likely using better strategies like block or filesystem cloning if you are doing VM hosting off a ZFS filesystem. Not doing so would be throwing away one of it's primary differentiators as a filesystem in such an environment. General purpose fileserving or personal desktop/laptop use generally has very few d…

> General purpose fileserving or personal desktop/laptop use generally has very few duplicated blocks and is not worth the overhead. I would contest this is because we don't have a good transparent deduplication right now - just some bad compromises. Hard copies? Edit anything and it gets edited everywhere - not what you want. Symlinks? Look different enough that programs treat them differently. I would argue your re…

But according to the article the regular desktop already has such a dedup system:

> If you put all this together, you end up in a place where so long as the client program (like /bin/cp) can issue the right copy offload call, and all the layers in between can translate it (eg the Window application does FSCTL_SRV_COPYCHUNK, which Samba converts to copy_file_range() and ships down to OpenZFS). And again, because there’s that clear and unambiguous signal that the data already exists and also it’s right there, OpenZFS can just bump the refcount in the BRT.

Re: OpenZFS deduplication is good now and you shouldn't use it

#153

Post by magicalhippo »

spockz wrote: post #108
AndrewDavis wrote: post #101

Earlier quoted context omitted.

So this is great, if you're just looking to deduplicate read only files. Less so if you intend to write to them. Write to one and they're both updated. Anyway. Offline/lazy dedup (not in the zfs dedup sense) is something that could be done in userspace, at the file level on any filesystem that supports reflinks. When a tool like rdfind finds a duplicate, instead of replacing with a hardlink, create a copy of the file…

How would this work if I have snapshots? Wouldn’t then the version of the file I just replaced still be in use there? But maybe I also need to store the copy again if I make another snapshot because the “original “ file isn’t part of the snapshot? So now I’m effectively storing more not less?

AFAIK, yes. Blocks are reference counted, so if the duplicate file is in a snapshot then the blocks would be referenced by the snapshot and hence not be eligible for deallocation. Only once the reference count falls to zero would the block be freed.

This is par for the course with ZFS though. If you delete a non-duplicated file you don't get the space back until any snapshots referencing the file are deleted.

Re: OpenZFS deduplication is good now and you shouldn't use it

#154

Post by pdimitar »

pezezin wrote: post #80
acdha wrote: post #37

Earlier quoted context omitted.

> Dedupe/compression works really well on syslog servers where I've seen 6:1 savings. Don’t you compress these directly? I normally see at least twice that for logs doing it at the process level.

Yes, that ratio is very small. I built a very simple, custom syslog solution, a syslog-ng server writing directly to a TimescaleDB hypertable ( https://www.timescale.com/ ) that is then presented as a Grafana dashboard, and I am getting a 30x compression ratio.

Would love to see your solution -- if it's open source.

Re: OpenZFS deduplication is good now and you shouldn't use it

#155

Post by Wowfunhappy »

Dylan16807 wrote: post #84
Wowfunhappy wrote: post #65

Earlier quoted context omitted.

You need block pointer rewrite for this?

You don't need it to dedup writable files. But redundant copies in snapshots are stuck there as far as I'm aware. So if you search for duplicates every once in a while, you're not going to reap the space savings until your snapshots fully rotate.

Thanks. I do think dedupe for non-snapshots would still be useful, since as you say most people will get rid of old snapshots eventually.

I also wonder if it would make sense for ZFS to always automatically dedupe before taking a snapshot. But you'd have to make this behavior configurable since it would turn shapshotting from a quick operation into an expensive one.

Re: OpenZFS deduplication is good now and you shouldn't use it

#156

Post by ThePowerOfFuet »

emptiestplace wrote: post #120
kderbe wrote: post #20

I clicked because of the bait-y title, but ended up reading pretty much the whole post, even though I have no reason to be interested in ZFS. (I skipped most of the stuff about logs...) Everything was explained clearly, I enjoyed the writing style, and the mobile CSS theme was particularly pleasing to my eyes. (It appears to be Pixyll theme with text set to the all-important #000, although I shouldn't derail this dis…

It scrolls horizontally :(

Not on Firefox on Android it doesn't.

Re: OpenZFS deduplication is good now and you shouldn't use it

#157

Post by burnt-resistor »

Already don't use ZoL because of their history of arms shrug-level support coupled with a lack of QA. ZoL != Solaris ZFS. It is mostly an aspirational cargo cult. Only a few fses like XFS and Ext4 have meaningful real-world, enterprise deployment hours. Technically, btrfs has significant (web ops instead of IT ops) deployment exposure due to its use on 10M boxes at Meta. Many non-mainstream fses also aren't assured to be trustworthy because of their low usage and prevalent lack of thorough, formalized QA. There's nothing wrong with experimentation, but it's necessary to have an accurate understanding of the risk budget for a given technology for a given use-case.

Re: OpenZFS deduplication is good now and you shouldn't use it

#158

Post by dspillett »

ThePowerOfFuet wrote: post #156
emptiestplace wrote: post #120

Earlier quoted context omitted.

It scrolls horizontally :(

Not on Firefox on Android it doesn't.

It does in chrome on android (1080 px wide screen, standard ppi & zoom levels) but not by enough that you see it on the main body text (scrolling just reveals more margin), so you might find it does for you too but not enough that you noticed.

As it is scrolling here, though inconsequentially, it might be bad on a smaller device with less screen and/or other ppi settings.

Re: OpenZFS deduplication is good now and you shouldn't use it

#159

Post by jamesbfb »

simonjgreen wrote: post #137

We used to make extensive use of, and gained huge benefit from, dedup in ZFS. The specific use case was storage for VMWare clusters where we had hundreds of Linux and Windows VMs that were largely the same content. [this was pre-Docker]

Can relate. I’ve recently taken ownership of a new work laptop with Ubuntu (with “experimental” zfs) and using dedupe on my nix store has been an absolute blessing!

Re: OpenZFS deduplication is good now and you shouldn't use it

#160

Post by Sakos »

LeoPanthera wrote: post #40
Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

btrfs has this. You can deduplicate a filesystem after the fact, as an overnight cron job or whatever. I really wish ZFS could do this.

This is my favourite hack for the Steam Deck. By switching my SD cards and the internal SSD to btrfs, the space savings are unreal (easily halving used space). Every game gets its own prefix which means a crazy amount of file duplication.
Post reply on HN