Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS deduplication is good now and you shouldn't use it

despairlabs.com

Post reply on HN
Print view View original

241–250 of 250 posts

Re: OpenZFS deduplication is good now and you shouldn't use it

#241

Post by magicalhippo »

spockz wrote: post #234
magicalhippo wrote: post #153

Earlier quoted context omitted.

AFAIK, yes. Blocks are reference counted, so if the duplicate file is in a snapshot then the blocks would be referenced by the snapshot and hence not be eligible for deallocation. Only once the reference count falls to zero would the block be freed. This is par for the course with ZFS though. If you delete a non-duplicated file you don't get the space back until any snapshots referencing the file are deleted.

Yes that snapshots incur a cost I know. But I’m wondering whether now the action of deduplicating actually created an extra copy instead of saving ’one.

I don't fully understand the scenario you mentioned. Could you perhaps explain in a bit more detail?

Re: OpenZFS deduplication is good now and you shouldn't use it

#242

Post by UltraSane »

Dylan16807 wrote: post #212
UltraSane wrote: post #174

Earlier quoted context omitted.

Windows NTFS has transparent file level compression that works quite well.

I don't know how much I agree with that. The old kind of NTFS compression from 1993 is completely transparent, but it uses a weak algorithm and processes each 64KB of file completely independently. It also fragments files to hell and back. The new kind from Windows 10 has a better algorithm and can have up to 2MB of context, which is quite reasonable. But it's not transparent to writes, only to reads. You have to man…

I'm talking about the "Compress contents to save disk space" option in the Advanced Attributes. It makes the file blue. I enable it on all .txt log files because it is so effective and completely transparent. It compresses a 25MB Google Drive log file to 8MB

Re: OpenZFS deduplication is good now and you shouldn't use it

#243

Post by Dylan16807 »

UltraSane wrote: post #242
Dylan16807 wrote: post #212

Earlier quoted context omitted.

I don't know how much I agree with that. The old kind of NTFS compression from 1993 is completely transparent, but it uses a weak algorithm and processes each 64KB of file completely independently. It also fragments files to hell and back. The new kind from Windows 10 has a better algorithm and can have up to 2MB of context, which is quite reasonable. But it's not transparent to writes, only to reads. You have to man…

I'm talking about the "Compress contents to save disk space" option in the Advanced Attributes. It makes the file blue. I enable it on all .txt log files because it is so effective and completely transparent. It compresses a 25MB Google Drive log file to 8MB

That's the old kind.

It's useful, but if they updated it it could get significantly better ratios and have less impact on performance.

Re: OpenZFS deduplication is good now and you shouldn't use it

#244

Post by girishso »

nikisweeting wrote: post #78
girishso wrote: post #72

Off topic, any tool to deduplicate files across different external Hard disks? Over the years I made multiple copies of my laptop HDD to different external HDDs, ended up with lots of duplicate copies of files.

How would you want the duplicates resolved? Just reported in some interface or would you want the duplicates deleted off some machines automatically? There are a few different ways you could solve it but it depends on what final outcome you need.

Just reporting in some plain text format so I can manually delete the duplicates, or create some script to delete.

I can't have like 10 external HDDs attached at the same time, so the tool needs to dump details (hashes?) somewhere on Mac HDD, and compare against those to find the duplicates.

Re: OpenZFS deduplication is good now and you shouldn't use it

#245

Post by ComputerGuru »

volkadav wrote: post #199
burnt-resistor wrote: post #157

Already don't use ZoL because of their history of arms shrug-level support coupled with a lack of QA. ZoL != Solaris ZFS. It is mostly an aspirational cargo cult. Only a few fses like XFS and Ext4 have meaningful real-world, enterprise deployment hours. Technically, btrfs has significant (web ops instead of IT ops) deployment exposure due to its use on 10M boxes at Meta. Many non-mainstream fses also aren't assured t…

I sympathize with your concerns for stability and testing, but I think that you might reconsider things in open-source ZFS land. OpenZFS/ZoL have been merged since the 2.0 release several years back, and some very large (e.g. Netflix) environments use FreeBSD which in turn uses OpenZFS, as well as being in use by the various Illumos derivatives and such. It is true that there has been some feature divergence between…

Don't disagree with your post but netflix doesn't use zfs for a couple of reasons, one of which is broken sendfile support (though that might be fixed soon!).

Re: OpenZFS deduplication is good now and you shouldn't use it

#246

Post by forrestthewoods »

fragmede wrote: post #218
forrestthewoods wrote: post #200

Earlier quoted context omitted.

Git LFS is a really really bad gross hack. It’s awful. https://www.forrestthewoods.com/blog/dependencies-belong-in-...

It's quite functional and usable, now. so I'd agree with hack, just not the rest of your adjectives. That was a good read! I've been thinking a lot about what comes after git too. One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo.

> One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo.

That’s the problem that a virtual file system solves. When you clone a repo it only materializes files when they’re accessed. This is how my work repo operates. It’s great.

Re: OpenZFS deduplication is good now and you shouldn't use it

#247

Post by dpedu »

UltraSane wrote: post #221
dpedu wrote: post #208

Earlier quoted context omitted.

> Linked clones aren't used in ESXi Huh? What do you mean? They absolutely are. I've made extensive use of them in ESXi/vsphere clusters in situations where I'm spinning up and down many temporary VMs.

Linked clones do not exist in ESXi. Horizon Composer is what is/was used to create them, and that requires a vCenter Server and a bit of infrastructure, including a database.

No, you can create them if you only have vCenter via its API. No extra infrastructure beyond that, though. The pyVmomi library has example code of how to do it. IIRC it is true that standalone ESXi does not offer the option to create a linked clone by itself, but if I wanted to be a pendant I'd argue that linked clones do exist in ESXi, as that is where vCenter deploys them.

Re: OpenZFS deduplication is good now and you shouldn't use it

#248

Post by fragmede »

forrestthewoods wrote: post #246
fragmede wrote: post #218

Earlier quoted context omitted.

It's quite functional and usable, now. so I'd agree with hack, just not the rest of your adjectives. That was a good read! I've been thinking a lot about what comes after git too. One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo.

> One thing you don't address is that no one wants all parts at once either, not would it fit on one computer, so I should be able to checkout just one subdirectory of a repo. That’s the problem that a virtual file system solves. When you clone a repo it only materializes files when they’re accessed. This is how my work repo operates. It’s great.

The problem I'm trying to avoid is having to dive down a hierarchy, and that not everyone needs or wants to know there is such a hierarchy. Like graphics design for foo-team only needs to have access to some subset of graphics. Arguably they could be given a symlink into the checkout, but the problem with that is of having a singular checkout.

The problem I had with Google3 is that the tools weren't great at branching and didn't fit my workflow, which tends to involve multiple checkouts (or worktrees using git). being forced to checkout the root of the repo, and then having to manage a symlink on top of that is no good for users that don't need/want to manage the complexity of having a single machine-global checkout.

Re: OpenZFS deduplication is good now and you shouldn't use it

#249

Post by nikisweeting »

girishso wrote: post #244
nikisweeting wrote: post #78

Earlier quoted context omitted.

How would you want the duplicates resolved? Just reported in some interface or would you want the duplicates deleted off some machines automatically? There are a few different ways you could solve it but it depends on what final outcome you need.

Just reporting in some plain text format so I can manually delete the duplicates, or create some script to delete. I can't have like 10 external HDDs attached at the same time, so the tool needs to dump details (hashes?) somewhere on Mac HDD, and compare against those to find the duplicates.

Here you go:

    cd /path/to/drive
    find . -type f -exec sha256sum {} + | sed -E 's/^([^ ]+) \./\1,/' >> ~/all_hashes.txt
Run that for each drive, then when you're done run:

    sort ~/all_hashes.txt > ~/sorted_hashes.txt
    awk -F, 'NR==1{print;next} {print $0 | "sort | uniq -w64 -D"}' ~/sorted_hashes.txt > ~/non_unique_hashes.txt
The output in ~/non_unique_hashes.txt will contain only the non-unique hashes that appear on more than one path.

Re: OpenZFS deduplication is good now and you shouldn't use it

#250

Post by nixdev »

Wowfunhappy wrote: post #5

I want "offline" dedupe, or "lazy" dedupe that doesn't require the pool to be fully offline, but doesn't happen immediately. Because: > When dedup is enabled [...] every single write and free operation requires a lookup and a then a write to the dedup table, regardless of whether or not the write or free proper was actually done by the pool. To me, this is "obviously" the wrong approach in most cases. When I'm writin…

You can already do offline/lazy dedupe.

  zfs set mountpoint=foopy/foo /mnt/foo
  zfs set dedup=off  foopy/foo

  zfs set mountpoint=foopy/baz /mnt/baz
  zfs set dedup=on   foopy/baz
Save all your stuff in /mnt/foo, then when you want to dedup do

  mv /mnt/foo/bar /mnt/baz/
Yeah... this feels like picrel, and it is

  https://i.pinimg.com/originals/cb/09/16/cb091697350736aae53afe4b548b9d43.jpg
but it's here and now and you can do it now.
Post reply on HN