Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

101–110 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#101

Post by AndrewDavis »

tiagod wrote: post #67
Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine! https://github.com/pauldreik/rdfind

So this is great, if you're just looking to deduplicate read only files. Less so if you intend to write to them. Write to one and they're both updated.

Anyway. Offline/lazy dedup (not in the zfs dedup sense) is something that could be done in userspace, at the file level on any filesystem that supports reflinks. When a tool like rdfind finds a duplicate, instead of replacing with a hardlink, create a copy of the file with `copy_file_range(2)` and let the filesystem create a reflink to it. Now you've got space savings and they're two separate files so if one is written to the other remains the same.

Re: OpenZFS deduplication is good now and you shouldn't use it

#102

Post by lazide »

E39M5S62 wrote: post #97
abrookewood wrote: post #88

Earlier quoted context omitted.

Yeah agreed, very closely related - even more so on ZFS where the compression (AFAIK) is on a block level rather than a file level.

ZFS compression is for sure at the block level - it's fully transparent to the userland tools.

It could be at a file level and still transparent to user land tools, FYI. Depending on what you mean by ‘file level’, I guess.

Re: OpenZFS deduplication is good now and you shouldn't use it

#103

Post by lazide »

Dylan16807 wrote: post #84
Wowfunhappy wrote: post #65

Earlier quoted context omitted.

You need block pointer rewrite for this?

You don't need it to dedup writable files. But redundant copies in snapshots are stuck there as far as I'm aware. So if you search for duplicates every once in a while, you're not going to reap the space savings until your snapshots fully rotate.

The issue with this, in my experience, is that at some point that pro (exactly, and literally, only one copy of a specific bit of data despite many apparent copies) can become a con if there is some data corruption somewhere.

Sometimes it can be a similar issue in some edge cases performance wise, but usually caching can address those problems.

Efficiency being the enemy of reliability, sometimes.

Re: OpenZFS deduplication is good now and you shouldn't use it

#104

Post by lazide »

UltraSane wrote: post #41
klysm wrote: post #11

I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.

Why is it a disaster and what would you replace it with? Is the AWS S3 style API an improvement?

It’s only a ‘disaster’ if you are using it exclusively programmatically and want to do special tuning.

File systems are pretty good if you have a mix of human and programmatic uses, especially when the programmatic cases are not very heavy duty.

The programmatic scenarios are often entirely human hostile, if you try to imagine what would be involved in actually using them. Like direct S3 access, for example.

Re: OpenZFS deduplication is good now and you shouldn't use it

#105

Post by spockz »

UltraSane wrote: post #82
wongarsu wrote: post #64

Earlier quoted context omitted.

Assuming something reasonable like 20TB Toshiba MG10 HDDs and 64GB DDR4 ECC RAM, quick googling suggests that 1TB of disk space uses about 0.2-0.4W of power (0.2 in idle, 0.4 while writing), 5GB of RAM about 0.3-0.5W. So your break even on power is a bit earlier depending on the access pattern, but in the same ball park.

What about rack space?

Not just rack space. At a certain amount of disks you also need to get a separate server (chassis + main board + cpu + ram) to host the disks. Maybe you need that for performance reasons any way. But saving disk space and only paying for it with some ram sounds cost effective.

Re: OpenZFS deduplication is good now and you shouldn't use it

#106

Post by doublepg23 »

eek2121 wrote: post #28

So many flaws. I want to see the author repeat this across 100TB of random data from multiple clients. He/she/whatever will quickly realize why this feature exists. One scenario I am aware of that uses another filesystem in a cloud setup saved 43% of disk space by using dedupe. No, you won't save much on a client system. That isn't what the feature is made for.

I assuming the author is aware why the feature exists since they state in the second sentence they funded the improvement over the course of two years?

Re: OpenZFS deduplication is good now and you shouldn't use it

#107

Post by spockz »

ShroudedNight wrote: post #81
tw04 wrote: post #69

Earlier quoted context omitted.

They are not the same thing, but when you boil it down to the raw math, they aren't identical twins, but they're absolutely fraternal twins. Both are trying to eliminate repeating data, it's just the frame of reference that changes. Compression in this context is operating on a given block or handful of blocks. Deduplication is operating on the entire "volume" of data. "Volume" having a different meaning depending on…

Is "paternal twins" a linguistic borrowing of some sort? It seems a relatively novel form of what I've mostly seen referred to as monozygotic / 'identical' twins. Searching for some kind of semi-canonical confirmation of its widespread use turns up one, maybe two articles where it's treated as an orthodox term, and at least an equal number of discussions admonishing its use.

If anything I would expect the term “maternal” twin to be used as whether or not a twin is monozygotic or “identical” depends on the amount of eggs from the mother.

Re: OpenZFS deduplication is good now and you shouldn't use it

#108

Post by spockz »

AndrewDavis wrote: post #101
tiagod wrote: post #67

Earlier quoted context omitted.

I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine! https://github.com/pauldreik/rdfind

So this is great, if you're just looking to deduplicate read only files. Less so if you intend to write to them. Write to one and they're both updated. Anyway. Offline/lazy dedup (not in the zfs dedup sense) is something that could be done in userspace, at the file level on any filesystem that supports reflinks. When a tool like rdfind finds a duplicate, instead of replacing with a hardlink, create a copy of the file…

How would this work if I have snapshots? Wouldn’t then the version of the file I just replaced still be in use there? But maybe I also need to store the copy again if I make another snapshot because the “original “ file isn’t part of the snapshot? So now I’m effectively storing more not less?

Re: OpenZFS deduplication is good now and you shouldn't use it

#109

Post by spockz »

UltraSane wrote: post #39
nisten wrote: post #36

can someone smarter than me explain what happens when instead of the regular 4kb block size in kernel builds we use 16kb or 64kb block size or is that only for the memory part, i am confused. Will a larger block size make this thing good or bad?

Generally the smaller the dedupe block the better as you are far more likely to find a matching block. But larger blocks will reduce the number of hashes you have to store. In my experience 4KB is the sweet spot to maximize how much data you save.

So in this case I think it would make sense to have a separate pool where you store large files like media so you can save on the dedup for them.

Is there an inherent performance loss of using 64kB blocks on FS level when using storage devices that are 4kB under the hood?

Re: OpenZFS deduplication is good now and you shouldn't use it

#110

Post by spockz »

tiffanyh wrote: post #44

OT: does anyone have a good way to dedupe iCloud Photos. Or my Dropbox photos?

Or better yet. A single photo I take of the kids will be stored in my camera roll. I will then share it with family using three different messengers. Now I have 4 copies. Each of the individual (recoded) are stored inside those messengers and also backed up. This even happens when sharing the same photo multiple times in different chats with the same messenger.

Is there any way to do de duplication here? Or just outright delete all the derivatives?

Post reply on HN