Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

21–30 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#21

Post by uniqueuid »

nikisweeting wrote: post #18
uniqueuid wrote: post #15

Earlier quoted context omitted.

I get the use case, but in most cases (and particularly this one) I'm sure it would be much better to implement that client-side. You may have seen in the WARC standard that they already do de-duplication based on hashes and use pointers after the first store. So this is exactly a case where FS-level dedup is not all that good.

WARC only does deduping within a single WARC, I'm talking about deduping across millions of WARCs.

That's not true, you commonly have CDX index files which allow for de-duplication across arbitrarily large archives. The internet archive could not reasonably operate without this level of abstraction.

[edit] Should add a link, this is a pretty good overview, but you can also look at implementations such as the new zeno crawler.

https://support.archive-it.org/hc/en-us/articles/208001016-A...

Re: OpenZFS deduplication is good now and you shouldn't use it

#22

Post by mappu »

nikisweeting wrote: post #2

I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools c…

Even with this change ZFS dedupe is still block-aligned, so it will not match repeated web assets well unless they exist at consistently identical offsets within the warc archives.

dm-vdo has the same behaviour.

You may be better off with long-range solid compression instead, or unpacking the warc files into a directory equivalent, or maybe there is some CDC-based FUSE system out there (Seafile perhaps)

Re: OpenZFS deduplication is good now and you shouldn't use it

#23

Post by nikisweeting »

uniqueuid wrote: post #21
nikisweeting wrote: post #18

Earlier quoted context omitted.

WARC only does deduping within a single WARC, I'm talking about deduping across millions of WARCs.

That's not true, you commonly have CDX index files which allow for de-duplication across arbitrarily large archives. The internet archive could not reasonably operate without this level of abstraction. [edit] Should add a link, this is a pretty good overview, but you can also look at implementations such as the new zeno crawler. https://support.archive-it.org/hc/en-us/articles/208001016-A...

Ah cool, TIL, thanks for the link. I didn't realize that was possible.

I know of the CDX index files produced by some tools but don't know anything about the details/that they could be used to dedup across WARCs, I've only been referencing the WARC file specs via IIPC's old standards docs.

Re: OpenZFS deduplication is good now and you shouldn't use it

#24

Post by nikisweeting »

mappu wrote: post #22
nikisweeting wrote: post #2

I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools c…

Even with this change ZFS dedupe is still block-aligned, so it will not match repeated web assets well unless they exist at consistently identical offsets within the warc archives. dm-vdo has the same behaviour. You may be better off with long-range solid compression instead, or unpacking the warc files into a directory equivalent, or maybe there is some CDC-based FUSE system out there (Seafile perhaps)

I should clarify I don't use WARCs at all with archivebox, it just stores raw files on the filsystem because I rely on ZFS for all my compression, so there is no offset alignment issue.

The wget extractor within archivebox can produce WARCs as an output but no parts of ArchiveBox are built to rely on those, they are just one of the optional extractors that can be run.

Re: OpenZFS deduplication is good now and you shouldn't use it

#25

Post by alchemist1e9 »

nikisweeting wrote: post #2

I'm so excited about fast dedup. I've been wanting to use ZFS deduping for ArchiveBox data for years, as I think fast dedup may finally make it viable to archive many millions of URLs in one collection and let the filesystem take care of compression across everything. So much of archive data is the same jquery.min.js, bootstrap.min.css, logo images, etc. repeated over and over in thousands of snapshots. Other tools c…

While a slightly different use case, I suspect you’d like zbackup if you don’t know about it.

Re: OpenZFS deduplication is good now and you shouldn't use it

#26

Post by phongn »

magicalhippo wrote: post #7
dark-star wrote: post #4

I wonder why they are having so much trouble getting this working properly with smaller RAM footprints. We have been using commercial storage appliances that have been able to do this for about a decade (at least) now, even on systems with "little" RAM (compared to the amount of disk storage attached). Just store fingerprints in a database and run through that at night and fixup the block pointers...

> and fixup the block pointers That's why. Due to reasons[1], ZFS does not have the capability to rewrite block pointers. It's been a long requested feature[2] as it would also allow for defragmentation. I've been thinking this could be solved using block pointer indirection, like virtual memory, at the cost of a bit of speed. But I'm by no means a ZFS developer, so there's surely something I'm missing. [1]: http://e…

It looks like they’re playing more with indirection features now (created for vdev removal) for other features. One of the recent summit hackathons sketched out using indirect vdevs to perform rebalancing.

Once you get a lot of snapshots, though, the indirection costs start to rise.

Re: OpenZFS deduplication is good now and you shouldn't use it

#28

Post by eek2121 »

So many flaws. I want to see the author repeat this across 100TB of random data from multiple clients. He/she/whatever will quickly realize why this feature exists. One scenario I am aware of that uses another filesystem in a cloud setup saved 43% of disk space by using dedupe.

No, you won't save much on a client system. That isn't what the feature is made for.

Re: OpenZFS deduplication is good now and you shouldn't use it

#29

Post by 0x457 »

tilt_error wrote: post #8

If writing performance is critical, why bother with deduplication at writing time? Do deduplication afterwards, concurrently and with lower priority?

Because to make this work without a lot of copying, you would need to mutate things that ZFS absolutely does not want to make mutable.

Re: OpenZFS deduplication is good now and you shouldn't use it

#30

Post by UltraSane »

"And this is the fundamental issue with traditional dedup: these overheads are so outrageous that you are unlikely to ever get them back except on rare and specific workloads."

This struck me as a very odd claim. I've worked with Pure and Dell/EMC arrays and for VMWare workloads they normally got at least 3:1 dedupe/compression savings. Only storing one copy of the base VM image works extremely well. Dedupe/compression works really well on syslog servers where I've seen 6:1 savings.

The effectiveness of dedupe is strongly affected by the size of the blocks being hashed, with the smaller the better. As the blocks get smaller the odds of having a matching block grow rapidly. In my experience 4KB is my preferred block size.

Post reply on HN