Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperspace

hypercritical.co

Post reply on HN
Print view View original

331–340 of 503 posts

Re: Hyperspace

#331

Post by gurjeet »

andrewla wrote: post #106
albertzeyer wrote: post #74

I wrote a similar (but simpler) script which would replace a file by a hardlink if it has the same content. My main motivation was for the packages of Python virtual envs, where I often have similar packages installed, and even if versions are different, many files would still match. Some of the packages are quite huge, e.g. Numpy, PyTorch, TensorFlow, etc. I got quite some disk space savings from this. https://githu…

This does not use hard links or symlinks; this uses a feature of the filesystem that allows the creation of copy-on-write clones. [1] [1] https://en.wikipedia.org/wiki/Apple_File_System#Clones

So albertzeyer's script can be adapted to use `cp -c` command, to achieve the same effect as Hyperspace.

Re: Hyperspace

#332

Post by mentalgear »

petercooper wrote: post #6

I love the model of it being free to scan and see if you'd get any benefit, then paying for the actual results. I, too, am a packrat, ran it, and got 7GB to reclaim. Not quite worth the squeeze for me, but I appreciate it existing!

am I really that old that I remember this being the default for most of the software about 10 years ago? Are people already that used to the subscription trap that they think this is a new model ?

Re: Hyperspace

#333

Post by mentalgear »

bob1029 wrote: post #48

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safel…

what's the source of that quote, does it mean it's not safe to use hyperspace ?

Re: Hyperspace

#334

Post by bsimpson »

_rend wrote: post #270
bsimpson wrote: post #161

Earlier quoted context omitted.

Which means if you actually edited those files, you might fill up your HD much more quickly than you expected. But if you have the same 500MB of node_modules in each of your dozen projects, this might actually durably save some space.

> Which means if you actually edited those files, you might fill up your HD much more quickly than you expected. I'm not sure if this is what you intended, but just to be sure: writing changes to a cloned file doesn't immediately duplicate the entire file again in order to write those changes — they're actually written out-of-line, and the identical blocks are only stored once. From [the docs](^1) posted in a sibling…

Thanks for the clarification!

Re: Hyperspace

#335

Post by tomnipotent »

eps wrote: post #319
Take8435 wrote: post #301

Downloaded. Ran it. Tells me "900" files can be cleaned. No summary, no list. But I was at least asked to buy the app. Why would I buy the app if I have no idea if it'll help?

This reminds me - Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs. They all inevitably found at least few KB to be reclaimed through their magic even if the same optimizer was run back to back with itself and allowed to "optimize" things. That is, on each run they always find extra memory to be freed. They ultimately did nothing but claim they did the wo…

I remember using MemTurbo in the Windows 2000 era, though now I know it was mostly smoke and mirrors. My biggest gripe these days is too many "hardware accelerated" apps eating away VRAM, which is less of a problem with Windows (better over-commit) but which causes me a few crashes a month on KDE.

Re: Hyperspace

#337

Post by jeromegv »

bhouston wrote: post #241
borland wrote: post #177

Earlier quoted context omitted.

He "only" saved 30%? That's amazing. I really doubt most people are going to get anywhere near that. When I run it on my home folder (Roughly 500GB of data) I find 124 MB of duplicated files. At this stage I'd like it to tell me what those files are - The dupes are probably dumb ones that I can simply go delete by hand, but I can understand why he'd want people to pay up first, as by simply telling me what the dupes…

> He "only" saved 30%? That's amazing. I really doubt most people are going to get anywhere near that. You misunderstood my comment. I ran it on my home folder which contains 165GB of data and it found 1.3GB is savings. That isn't significant for me to care about because I currently have 225GB free of my 512GB drive. BTW I highly recommend the free "disk-inventory-x" utility for MacOS space management.

Everyone misunderstood your comment for a reason.

You wrote: but it only found 1GB of savings on a 8.1GB folder.

It’s quite a saving and that’s what everyone understood from your comment.

Re: Hyperspace

#338

Post by pmarreck »

abrookewood wrote: post #284
pmarreck wrote: post #254

Earlier quoted context omitted.

It's considered an "expensive" configuration that is only good for certain use-cases, though, due to its memory requirements.

Yes true, but that page also covers some recent improvements to de-duplication that might assist.

Really? I haven't looked at this ZFS feature in a few years so I will take a look

EDIT: Is this referring to the "fast" dedup feature?

Re: Hyperspace

#339

Post by elicash »

Take8435 wrote: post #301

Downloaded. Ran it. Tells me "900" files can be cleaned. No summary, no list. But I was at least asked to buy the app. Why would I buy the app if I have no idea if it'll help?

It didn't tell you how much disk space? It's supposed to. It only told you the number of files?

Re: Hyperspace

#340

Post by a-dub »

bob1029 wrote: post #48

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safel…

a content addressed block store with pointers and skiplists for file continuity would be kinda neat.
Post reply on HN