Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperspace

hypercritical.co

Post reply on HN
Print view View original

211–220 of 503 posts

Re: Hyperspace

#211

Post by dewey »

augusto-moura wrote: post #203
amelius wrote: post #192

Earlier quoted context omitted.

There are several such tools for Linux, and they are free, so maybe just change operating systems.

I'm pretty sure some of them also work on MacOS. rmlint[1], for example can output a script that reflinks duplicates (or run any script for both files): rmlint -c sh:handler=reflink . I'm not sure if reflink works out of the box, but you can write your own alternative script that just links both files [1]: https://github.com/sahib/rmlint

It does not support APFS: https://github.com/sahib/rmlint/issues/421

Re: Hyperspace

#212

Post by wlesieutre »

borland wrote: post #177
horsawlarway wrote: post #116

Earlier quoted context omitted.

He picked node_modules because it's highly likely to encounter redundant files there. If you read the rest of the comment he only saved another 30% running his entire user home directory through it. So this is not a linear trend based on space used.

He "only" saved 30%? That's amazing. I really doubt most people are going to get anywhere near that. When I run it on my home folder (Roughly 500GB of data) I find 124 MB of duplicated files. At this stage I'd like it to tell me what those files are - The dupes are probably dumb ones that I can simply go delete by hand, but I can understand why he'd want people to pay up first, as by simply telling me what the dupes…

Another 30% more than the 1GB saved in node modules, for 1.3GB total. Not 30% of total disk space.

For reference, from the comment they’re talking about:

> I then tried again including my user home folder (731K files, 127K folders, 2755 eligible files) to hopefully catch more savings and I only ended up at 1.3GB of savings (300MB more than just what was in the NodeJS folders.)

Re: Hyperspace

#213

Post by sgerenser »

f1shy wrote: post #185
borland wrote: post #158

Earlier quoted context omitted.

I don't know exactly what Siracusa is doing here, but I can take an educated guess: For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives. The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not t…

I think the prob. is not so low. I remember reading here about a person getting a foto of another chat in a chat application, which was using sha in the background. I do not recall all the details, it is improbable, but possible.

LOL nope, I seriously doubt that was the result of a SHA256 collision.

Re: Hyperspace

#214

Post by throwanem »

furyofantares wrote: post #174
radlad wrote: post #162

Earlier quoted context omitted.

I'm a bit confused as the Mac App Store says it's over 4 years old.

The 4+ Age rating is like, who can use the app. Not for 3 year olds, apparently.

I feel like that's true for most of the relatively low-level disk and partition management tooling. As unpopular an opinion as it may lately be around here, I'm enough of a pedagogical traditionalist to remain convinced that introductory logical volume management is best left at least till kindergarten.

Re: Hyperspace

#215

Post by dewey »

DontBreakAlex wrote: post #181

Nice, but I'm not getting a subscription for a filesystem utility. Had it been a one-time $5 license, I would have bought it. At the current price, it's literally cheaper to put files in a S3 bucket or outright buy an SSD.

> Two kinds of purchases are possible: one-time purchases and subscriptions.

https://hypercritical.co/hyperspace/#purchase

Re: Hyperspace

#216

Post by williamsmj »

astennumero wrote: post #149

What algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?

Deleted comment based on a misunderstanding.

Re: Hyperspace

#217

Post by rzzzt »

borland wrote: post #158
astennumero wrote: post #149

What algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?

I don't know exactly what Siracusa is doing here, but I can take an educated guess: For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives. The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not t…

I experimented with a similar, "hardlink farm"-style approach for deduplicated, browseable snapshots. It resulted in a small bash script which did the following:

- compute SHA256 hashes for each file on the source side

- copy files which are not already known to a "canonical copies" folder on the destination (this step uses the hash itself as the file name, which makes it easy to check if I had a copy from the same file earlier)

- mirror the source directory structure to the destination

- create hardlinks in the destination directory structure for each source file; these should use the original file name but point to the canonical copy.

Then I got too scared to actually use it :)

Re: Hyperspace

#218

Post by dewey »

For those mentioning that there's no price listed, it's not that easy as in the App Store the price varies by country. You can open the App Store link and then look at "In App Purchases" though.

For me on the German store it looks like this:

    Unlock for One Year 22,99 €
    Unlock for One Month 9,99 €
    Lifetime Unlock 59,99 €
So it supports both one time purchases and subscriptions. Depending on what you prefer. More about that here: https://hypercritical.co/hyperspace/#purchase
Post reply on HN