Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

81–90 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#81

Post by ShroudedNight »

tw04 wrote: post #69
wmf wrote: post #59

Earlier quoted context omitted.

VMs are known to benefit from dedupe so yes, you'll see benefits there. ZFS is a general-purpose filesystem not just an enterprise SAN so many ZFS users aren't running VMs. Dedupe/compression works really well on syslog I apologize for the pedantry but dedupe and compression aren't the same thing (although they tend to be bundled in the enterprise storage world). Logs are probably benefiting from compression not dedu…

They are not the same thing, but when you boil it down to the raw math, they aren't identical twins, but they're absolutely fraternal twins. Both are trying to eliminate repeating data, it's just the frame of reference that changes. Compression in this context is operating on a given block or handful of blocks. Deduplication is operating on the entire "volume" of data. "Volume" having a different meaning depending on…

Is "paternal twins" a linguistic borrowing of some sort? It seems a relatively novel form of what I've mostly seen referred to as monozygotic / 'identical' twins. Searching for some kind of semi-canonical confirmation of its widespread use turns up one, maybe two articles where it's treated as an orthodox term, and at least an equal number of discussions admonishing its use.

Re: OpenZFS deduplication is good now and you shouldn't use it

#82

Post by UltraSane »

wongarsu wrote: post #64
floating-io wrote: post #56

Earlier quoted context omitted.

That doesn't account for OpEx, though, such as power...

Assuming something reasonable like 20TB Toshiba MG10 HDDs and 64GB DDR4 ECC RAM, quick googling suggests that 1TB of disk space uses about 0.2-0.4W of power (0.2 in idle, 0.4 while writing), 5GB of RAM about 0.3-0.5W. So your break even on power is a bit earlier depending on the access pattern, but in the same ball park.

What about rack space?

Re: OpenZFS deduplication is good now and you shouldn't use it

#84

Post by Dylan16807 »

Wowfunhappy wrote: post #65
Dylan16807 wrote: post #12

Earlier quoted context omitted.

The ability to alter existing snapshots, even in ways that fully preserve the data, is extremely limited in ZFS. So yes that would be great, but if I was holding my breath for Block Pointer Rewrite I'd be long dead.

You need block pointer rewrite for this?

You don't need it to dedup writable files. But redundant copies in snapshots are stuck there as far as I'm aware. So if you search for duplicates every once in a while, you're not going to reap the space savings until your snapshots fully rotate.

Re: OpenZFS deduplication is good now and you shouldn't use it

#85

Post by cryptonector »

Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

Lazy/off-line dedup requires block pointer rewrite, but ZFS _cannot_ and will not ever get true BP rewrite because ZFS is not truly a CAS system. The problem is that physical locations are hashed into the Merkle hash tree, and that makes moving physical locations prohibitively expensive as you have to rewrite all the interior nodes on the way to the nodes you want to rewrite.

A better design would have been to split every node that has block pointers into two sections, one that has only logical block pointers and all of whose contents gets hashed into the tree, and one that has only the physical locations (as if it were a cache) of the corresponding logical block pointers in the first section, with the second section _not_ hashed into the Merkle hash tree. Then BP rewrite would only require re-writing blocks that are not part of the Merkle hash tree.

But as it is you can't get BP rewrite to work on ZFS, so you can't get what you're asking for.

Well... maybe. Perhaps on read hash mismatch ZFS could attempt to locate the pointed-to block in the dedup table using the hash from the pointer. Then ZFS could reallocate the dedup'ed block. The price you'd pay then is one pointless read -- not too bad. The impossibility of BP rewrite generally leads to band-aids like this.

Re: OpenZFS deduplication is good now and you shouldn't use it

#86

Post by Dylan16807 »

Wowfunhappy wrote: post #70
tiagod wrote: post #67

Earlier quoted context omitted.

I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine! https://github.com/pauldreik/rdfind

But then you have to be careful not to remove the one which happens to be the "original" or the hardlinks will break, right?

No, pointing to an original is how soft links work.

Hard links are all equivalent. A file has any number of hard links, and at least in theory you can't distinguish between them.

The risk with hardlinks is that you might alter the file. Reflinks remove that risk, and also perform very well.

Re: OpenZFS deduplication is good now and you shouldn't use it

#87

Post by AndrewDavis »

UltraSane wrote: post #51
wongarsu wrote: post #50

Earlier quoted context omitted.

I haven't tried it myself, but the widely quoted number for old ZFS dedup is that you need 5GB of RAM for every 1TB of disk space. Considering that 1 TB of disk space currently costs about $15 and 5GB of server RAM about $25, you need a 3:1 dedupe ratio just to break even. If your data is a good fit you might get away with 1GB per TB, but if you are out of luck the 5GB might not even be enough. That's why the article…

Why does it need so much RAM? It should only need to store the block hashes which should not need anywhere near that much RAM. Inline dedupe is pretty much standard on high-end storage arrays nowadays.

The linked blog post covers this, and the improvements made to make the new dedup better.

Re: OpenZFS deduplication is good now and you shouldn't use it

#88

Post by abrookewood »

UltraSane wrote: post #73
abrookewood wrote: post #68

Earlier quoted context omitted.

Couple of comments. Firstly, you are talking about highly redundant information when referencing VM images (e.g. the C drive on all Windows Serer images will be virtually identical), whereas he was using his own laptop contents as an example. Secondly, I think you are conflating two different features: compression & de-duplication. In ZFS you can have compression turned on (almost always worth it) for a pool, but sti…

Fair point. My experience is with enterprise storage arrays and I have always used dedupe/compression at the same time. Dedupe is going to be a lot less useful on single computers. I consider dedupe/compression to be two different forms of the same thing. compression reduces short range duplication while deduplication reduces long range duplication of data.

Yeah agreed, very closely related - even more so on ZFS where the compression (AFAIK) is on a block level rather than a file level.

Re: OpenZFS deduplication is good now and you shouldn't use it

#89

Post by mappu »

UltraSane wrote: post #41
klysm wrote: post #11

I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.

Why is it a disaster and what would you replace it with? Is the AWS S3 style API an improvement?

High-density drives are usually zoned storage, and it's pretty difficult to implement the regular filesystem API on top of that with any kind of reasonable performance (device- vs host- managed SMR). The S3 API can work great on zones, but only because it doesn't let you modify an existing object without rewriting the whole thing, which is an extremely rough tradeoff.

Re: OpenZFS deduplication is good now and you shouldn't use it

#90

Post by fragmede »

forrestthewoods wrote: post #63

My dream Git successor would use either dedupe or a simple cache plus copy-on-write so that repos can commit toolchains and dependencies and users wouldn’t need to worry about disk drive bloat. Maybe someday…

It does dedup using Sha-1 on entire files. you might try git-lfs for your usecase though.
Post reply on HN