Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperspace

hypercritical.co

Post reply on HN
Print view View original

281–290 of 503 posts

Re: Hyperspace

#281

Post by MBCook »

mintplant wrote: post #257
MBCook wrote: post #190

Earlier quoted context omitted.

APFS is a copy on write filesystem if you use the right APIs, so it does what you describe but only for entire files. I believe as soon as you change a single bite you get a complete copy that’s your own. And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.

I suppose this means that you could find yourself unexpectedly out of disk space in unintuitive ways, if you're only trying to change one byte in a cloned file but there isn't enough space to copy its entire contents?

I’m not sure if it works on a file or block level for CoW, but yes.

However APFS gives you a number of space related foot-guns if you want. You can overcommit partitions, for example.

It also means if you have 30 GB of files on disk that could take up anywhere from a few hundred K to 30 GB of actual data depending on how many dupes you have.

It’s a crazy world, but it provides some nice features.

Re: Hyperspace

#282

Post by abrookewood »

amzin wrote: post #146
taneliv wrote: post #88

Earlier quoted context omitted.

On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot d…

Is there a FS that keeps only diffs in clone files? It would be neat

ZFS: "The main benefit of deduplication is that, where appropriate, it can greatly reduce the size of a pool and the disk count and cost. For example, if a server stores files with identical blocks, it could store thousands or even millions of copies for almost no extra disk space." (emphasis added)

https://www.truenas.com/docs/references/zfsdeduplication/

Re: Hyperspace

#283

Post by abrookewood »

rappatic wrote: post #160
amzin wrote: post #146

Earlier quoted context omitted.

Is there a FS that keeps only diffs in clone files? It would be neat

I wondered that too. If we only have two files, A and its duplicate B with some changes as a diff, this works pretty well. Even if the user deletes A, the OS could just apply the diff to the file on disk, unlink A, and assign B to that file. But if we have A and two different diffs B1 and B2, then try to delete A, it gets a little murkier. Either you do the above process and recalculate the diff for B2 to make it a d…

ZFS does this by de-duplicating at the block level, not the file level. It means you can do what you want without needing to keep track of a chain of differences between files. Note that de-duplication on ZFS has had issues in the past, so there is definitely a trade-off. A newer version of de-duplication sounds interesting, but I don't have any experience with it: https://www.truenas.com/docs/references/zfsdeduplication/

Re: Hyperspace

#284

Post by abrookewood »

pmarreck wrote: post #254
albertzeyer wrote: post #76

Earlier quoted context omitted.

> This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. I think that ZFS actually does this. https://www.truenas.com/docs/references/zfsdeduplication/

It's considered an "expensive" configuration that is only good for certain use-cases, though, due to its memory requirements.

Yes true, but that page also covers some recent improvements to de-duplication that might assist.

Re: Hyperspace

#285

Post by karparov »

TL;DR: He wrote an OS X dedup app which finds files with the same contents and tells the filesystem that their contents are identical, so it can save space (using copy-on-write features).

He points out its dangerous but could be worth it cause space savings.

I wonder if the implementation is using a hash only or does an additional step to actually compare the contents to avoid hash collision issues.

It's not open source, so we'll never know. He chose a pay model instead.

Also, some files might not be identical but have identical blocks. Something that could be explored too. Other filesystems have that either in their tooling or do it online or both.

Re: Hyperspace

#286

Post by nielsbot »

bob1029 wrote: post #48

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safel…

Disk Utility.app manages to keep the OS running while make the disk exclusive-access.. I wonder how it does that.

Re: Hyperspace

#289

Post by criddell »

dylan604 wrote: post #130
pizzafeelsright wrote: post #67

Earlier quoted context omitted.

data loss is the largest concern I still do not trust de-duplication software.

Even using sha-256 or greater type of hashing, I'd still have concerns about letting a system make deletion decisions without my involvement. I've even been part of de-dupe efforts, so maybe my hesitation is just because I wrote some of the code and I know I'm not perfect in my coding or even my algo decision trees. I know that any mistake I made would not be of malice but just ignorance or other stupid mistake. I've…

> I'd still have concerns about letting a system make deletion decisions without my involvement

You are involved. You see the list of duplicates and can review them as carefully as you'd like before hitting the button to write the changes.

Re: Hyperspace

#290

Post by criddell »

jbverschoor wrote: post #46

Does it preserve all metadata, extended attributes, and alternate streams/named forks?

The FAQ talks about this a little:

Q: Does Hyperspace preserve file metadata during reclamation?

A: When Hyperspace replaces a file with a space-saving clone, it attempts to preserve all metadata associated with that file. This includes the creation date, modification date, permissions, ownership, Finder labels, Finder comments, whether or not the file name extension is visible, and even resource forks. If the attempt to preserve any of these piece of metadata fails, then the file is not replaced.

If you find some piece of file metadata that is not preserved, please let us know.

Q: How does Hyperspace handle resource forks?

A: Hyperspace considers the contents of a file’s resource fork to be part of the file’s data. Two files are considered identical only if their data and resource forks are identical to each other.

When a file is replaced by a space-saving clone during reclamation, its resource fork is preserved.

Post reply on HN