Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperspace

hypercritical.co

Post reply on HN
Print view View original

361–370 of 503 posts

Re: Hyperspace

#361

Post by pfranz »

DontBreakAlex wrote: post #316
benced wrote: post #191

Earlier quoted context omitted.

"I don't value software but that's not a respectable opinion so I'll launder that opinion via subscriptions"

Well I do value software, I'm paid $86/h to write some! I just find that for $20/year or $50 one time, you can get way more than 12G of hard drive space. I also don't think that this piece of software requires so much maintenance that it wouldn't be worth making at a lower price. I'm not saying that it's bad software, it's really great, just too expensive... Personally, my gut feeling is that the dev would have had m…

The first option presented is a one month non-renewing subscription for $10. I think the intention is periodically (once a year, once every few years?) you run it to reclaim space. If it was reclaiming more than a few gigs I would do it.

The author talked about being very conservative on launch; skipping directories like the Photo library or others apps that actively manage data or looking across user directories. He stumbled into writing this app because he noticed the duplicated data of shared Photo libraries between different users on the same machine. That use case isn't even supported in this version. He said he plans future development to safely dedup more data--making a one time purchase less sustainable for them.

Re: Hyperspace

#362

Post by jonhohle »

LVB wrote: post #349
jonhohle wrote: post #341

I made a command line utility called `dedup` a while back to do the same thing. It has a dry-run mode, will “intelligently” choose the best clone source, understands hard links and other clones, preserves metadata, deals with HFS compressed files properly. It hasn’t destroyed any of my own data, but like any file system tool, use at your own risk. 0 - https://github.com/ttkb-oss/dedup

Just tried it, and it works well! I didn't realize the potential of this technique until I saw just how many dupes there were of certain types of files, especially in node_modules. It wasn't uncommon to see it replace 50 copies of some js file with one, and that was just in a specific subdirectory. I see it is "pre-release" and sort of low GH stars (== usage?), so I'm curious about the stability since this type of to…

I use it on my family photos, work documents, etc. and have not run into an issue that I haven’t added a test for. I didn’t commercialize it because I didn’t know what I didn’t know, but the utility does try to fail quickly if any of the file system operations fail (cloning, metadata duplication, atomic swaps, etc.).

Whenever using it on something sensitive that I can’t back up first for whatever reason, I make checksum files and compare them afterwards. I’ve done this many times on hundreds of GB and haven’t seen corruption. Caveat emptor.

There is one huge caveat I should add to the README - block corruption happens. Having a second copy of a file is a crude form of backup. Cloning causes all instances to use the same block, so if that one instance is corrupted, all clones are. That’s fine for software projects with generated files that can be rebuilt or checked out again, but introduces some risk for files that may not otherwise be replaceable. I keep multiple backups of all that stuff in hardware other than where I’m deduping, so I dedup with abandon.

I’m a nobody with no audience. Maybe some attention here will get some users.

Re: Hyperspace

#363

Post by jonhohle »

pansa777 wrote: post #262
mintplant wrote: post #257

Earlier quoted context omitted.

I suppose this means that you could find yourself unexpectedly out of disk space in unintuitive ways, if you're only trying to change one byte in a cloned file but there isn't enough space to copy its entire contents?

It doesn't work like you think. If you change one byte of duplicated file - the only "byte" will be changed on disk (a "byte", because, technically is not a byte, but a block). As far as I understand, it works like a reflink feature in the modern linux FSs. If so, thats really cool, and thats also a bit better than the zfs's snapshots. Iam newbie on macos, but it looks amazing

That’s true as long as the writing application only writes to blocks that have changed. Is a VM tool going to write blocks, or write multi-MB segments of a sparse image that can be swapped atomically? Unfortunately, once a file changes there are no APIs to check which blocks are still shared (at least there weren’t as of macOS 13).

Re: Hyperspace

#364

Post by jonhohle »

_rend wrote: post #270
bsimpson wrote: post #161

Earlier quoted context omitted.

Which means if you actually edited those files, you might fill up your HD much more quickly than you expected. But if you have the same 500MB of node_modules in each of your dozen projects, this might actually durably save some space.

> Which means if you actually edited those files, you might fill up your HD much more quickly than you expected. I'm not sure if this is what you intended, but just to be sure: writing changes to a cloned file doesn't immediately duplicate the entire file again in order to write those changes — they're actually written out-of-line, and the identical blocks are only stored once. From [the docs](^1) posted in a sibling…

The key is “unmodified” and how APFS knows or doesn’t know whether they are modified. How many apps write on block boundaries or even mutate just in disk data that has changed vs overwriting or replacing atomically? For most applications there is no benefit and a significant risk of corruption.

So APFS supports it, but there is no way to control what an app is going to do, and after it’s done it, no way to know what APFS has done.

Re: Hyperspace

#365

Post by [deleted] »

WarOnPrivacy wrote: post #350
eps wrote: post #319

Earlier quoted context omitted.

This reminds me - Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs. They all inevitably found at least few KB to be reclaimed through their magic even if the same optimizer was run back to back with itself and allowed to "optimize" things. That is, on each run they always find extra memory to be freed. They ultimately did nothing but claim they did the wo…

> Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs. They ultimately did nothing but claim they did the work. Must've sold pretty well nonetheless. QEMM worked by remapping stuff into extended memory - in a time that most software wasn't interested in using it. It worked as advertised. Quarterdeck made good stuff all around. Desq and DesqView/X were amazin…

[deleted]

Re: Hyperspace

#366

Post by xp84 »

lukeholder wrote: post #344
borland wrote: post #274

Earlier quoted context omitted.

Now you understand why this app costs more than 2x the price of alternatives such as diskDedupe. Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no. The author - John Siracusa - has been a profes…

Best course of action is to not trust John, and just wait for a year of the app out the wild, until everyone else trusts John . I have enough hard drive space in the meantime to not rush into trusting John.

Having listened to John for 10 years, he’d be the first to encourage you to wait around to trust his app.

Re: Hyperspace

#368

Post by xp84 »

glitchc wrote: post #367

This is cool! Wait a minute, what happens to copies on different physical drives. Are they cloned too?

This operates within one drive. Possibly within one magical APFS “partition” (or whatever APFS calls partitions), I can’t remember

Re: Hyperspace

#369

Post by czk »

pizzafeelsright wrote: post #67
bob1029 wrote: post #48

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safel…

data loss is the largest concern I still do not trust de-duplication software.

Dedupe seemed more interesting when storage was expensive, but nowadays it feels like the overhead you get from running dedupe, in most cases, is priced-in. At least with software like CommVault for backups, dedupe requires beefy hardware and low-latency SSDs for the database, If there is even a few extra milliseconds of latency or the server can’t handle requests fast enough, your backup throughput absolutely tanks. Depending on your data though you could see some ridiculous savings here that make it worth the trouble.

I’ve heard many horror stories of dedupe related corruption or restoration woes though, especially after a ransomware attack.

Re: Hyperspace

#370

Post by xp84 »

jedbrooke wrote: post #291
petercooper wrote: post #6

I love the model of it being free to scan and see if you'd get any benefit, then paying for the actual results. I, too, am a packrat, ran it, and got 7GB to reclaim. Not quite worth the squeeze for me, but I appreciate it existing!

it’s very refreshing compared to those “free trials” you have to remember to cancel (pro tip: use virtual credit cards which you can lock for those so if you forget to cancel the charges are blocked) however has anyone been able to find out from the website how much the license actually costs?

Doesn’t the Mac App Store listing list the IAP SKUs like it does on iOS?
Post reply on HN