Live data from Hacker News

Hyperspace

hypercritical.co

111–120 of 503 posts

Re: Hyperspace

#111
Many comments here offering similar solutions based on hardlinks or symlinks.

This uses a specific feature of APFS that allows the creation of copy-on-write clones. [1] If a clone is written to, then it is copied on demand and the original file is unmodified. This is distinct from the behavior of hardlinks or symlinks.

[1] https://en.wikipedia.org/wiki/Apple_File_System#Clones

Re: Hyperspace

#112
post #88
post #48

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safel…

On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot d…

Files are always represented as lists of blocks or block spans within a file system. Individual blocks could in theory be partially shared between files at the complexity cost of a reference counter for each block. So changing a single byte in a copy on write file could take the same time regardless of file size because only the affected bock would have to be duplicated. I don't know at all how MacOS implements this copynon write scheme, though.

Re: Hyperspace

#113

A 20$ 1 year licence for something that probably has a FOSS equivalent on Linux... However, considering Apple will never ever ever allow user replaceable storage on a laptop, this might be worth it.

I have yet to see a GUI variant of deduplication software for Linux. There are plenty of command line tools, which probably can be ported to macOS, but there's no user friendly tool to just click through as far as I know. There's value in convenience. I wouldn't pay for a yearly license (that price seems more than fair for a "version lifetime" price to me?) but seeing as this tool will probably need constant maintena…

50$ for a lifetime license.

Which really means up until the dev gets bored, which can be as short as 18 months.

I wouldn't mind something like this versioned to OS. 20$ for the current OS, and ten dollars for every significant update.

Re: Hyperspace

#114

Earlier quoted context omitted.

No because it isn't getting rid of the duplicate, it's using a feature of APFS that allows for duplicates to exist separately but share the same internal data.

Is it not the same as a hard link (which I believe are supported on Mac too)?

Almost, but the difference is that if you change one of hardlinked files, you change "all of them". (It's really the same file but with different paths.)

https://hypercritical.co/hyperspace/#how-it-works

APFS apparently allows for creating "link files" which when changed, start to diverge.

Re: Hyperspace

#115
post #90

Earlier quoted context omitted.

No because it isn't getting rid of the duplicate, it's using a feature of APFS that allows for duplicates to exist separately but share the same internal data.

Right, but the concept is the same, "remove duplicates" in order to save storage space. If it's using reflinks, softlinks, APFS clones or whatever is more or less an implementation detail. I know that internally it isn't actually "removing" anything, and that it uses fancy new technology from Apple. But in order to explain the project to strangers, I think my tagline gets the point across pretty well.

> Right, but the concept is the same, "remove duplicates" in order to save storage space.

The duplicates aren't removed, though. Nothing changes from the POV of users or software that use those files, and you can continue to make changes to them independently.

Re: Hyperspace

#116

Earlier quoted context omitted.

The relevant number (missing from above) is the total amount of space on that storage device. If it saves 1GB on a 8TB drive, it's not a big win.

It should be proportional to the total used space, not the space available. The previous commenter said it was a 1 GB savings from ~8 GB of used space; that's equally significant whether it happens on a 10 GB drive or a 10 TB one.

He picked node_modules because it's highly likely to encounter redundant files there.

If you read the rest of the comment he only saved another 30% running his entire user home directory through it.

So this is not a linear trend based on space used.

Re: Hyperspace

#117
post #80
post #69

[flagged]

It is not a hard link. A clone is an independent file which is backed by the same storage. So far mostly the same as a hard link you’ll say. However if you modify a clone, it will be “uncloned” and will be modified independently of its clones.

And I believe that only the modified blocks are “uncloned”, the rest of the data is still shared between the two files

Re: Hyperspace

#120
post #44

Earlier quoted context omitted.

> it only found 1GB of savings on a 8.1GB folder. You "only" found that 12% of the space you are using is wasted? Am I reading this right?

The relevant number (missing from above) is the total amount of space on that storage device. If it saves 1GB on a 8TB drive, it's not a big win.

Absolutely, 100% backwards. The tool cannot save space from disk space that is not scanned. Your "not a big win" comment assumes that there is no space left to be reclaimed on the rest of the disk. Or that the disk is not empty, or that the rest of the disk can't be reclaimed at an even higher rate.
Post reply on HN