Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperspace

hypercritical.co

Post reply on HN
Print view View original

151–160 of 503 posts

Re: Hyperspace

#151

Post by diegs »

astennumero wrote: post #149

What algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?

This reminds me of https://en.wikipedia.org/wiki/Venti_(software) which was a content-addressible filesystem which used hashes for de-duplication. Since the hashes were computed at write time, the performance penalty is amortized.

Re: Hyperspace

#152

Post by madeofpalk »

BWStearns wrote: post #142

I have file A that's in two places and I run this. I modify A_0. Does this modify A_1 as well or just kind of reify the new state of A_0 while leaving A_1 untouched?

It's called copy-on-write because when you modify A_0, it'll make a copy of the file if you write to it but not A_1.

https://en.wikipedia.org/wiki/Copy-on-write#In_computer_stor...

Re: Hyperspace

#153

Post by w4yai »

astennumero wrote: post #149

What algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?

I'd hash the first 1024 bytes of all files, and starts from there is any collision. That way you don't need to hash the whole (large) files, but only those with same hashes.

Re: Hyperspace

#154

Post by kstrauser »

egorfine wrote: post #40
apercu wrote: post #25

Earlier quoted context omitted.

[flagged]

A ~20 y.o. account with perhaps hundreds of devices in history across different continents and countries along with family sharing. Every time I need to purchase something via Apple, it becomes a quest. Enter password, validate card, welcome to endless login loop. Reboot. Click purchase, enter password, confirm OTP on another device, then nothing happens, purchase button is active, clicks ignored. Reboot. Click "Get"…

I would never dismiss such a complaint with a glib "works for me". And yet, your experience is so utterly, completely different from mine that I have to think something's busted in your account somewhere. I've had an account for about as long, with family sharing and all the rest. I never, ever, have anywhere near that level of difficulty. For me it works as documented: I click "Get", it asks for Face ID to confirm it's really me, then a few seconds later I have the app installed and ready to use.

Again, I don't think you're doing anything wrong, and I don't doubt your experience. But I really think something's fundamentally wrong somewhere, because what you're dealing with is not normal. It's not the common experience others are tolerating.

Re: Hyperspace

#155

Post by lgdskhglsa »

BWStearns wrote: post #142

I have file A that's in two places and I run this. I modify A_0. Does this modify A_1 as well or just kind of reify the new state of A_0 while leaving A_1 untouched?

He's using the "copy on write" feature of the file system. So it should leave A_1 untouched, creating a new copy for A_0's modifications. More info: https://developer.apple.com/documentation/foundation/file_sy...

Re: Hyperspace

#156

Post by kstrauser »

w4yai wrote: post #153
astennumero wrote: post #149

What algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?

I'd hash the first 1024 bytes of all files, and starts from there is any collision. That way you don't need to hash the whole (large) files, but only those with same hashes.

At that point, why hash them instead of just using the first 1024 bytes as-is?

Re: Hyperspace

#157

Post by ryandrake »

david_allison wrote: post #136

> Hyperspace can’t be installed on “Macintosh HD” because macOS version 15 or later is required. macOS 15 was released in September 2024, this feels far too soon to deprecate older versions.

Came here to post the same thing. Would love to try the application, but I guess not if the developer is deliberately excluding my device (which cannot run the bleeding edge OS).

Re: Hyperspace

#158

Post by borland »

astennumero wrote: post #149

What algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?

I don't know exactly what Siracusa is doing here, but I can take an educated guess:

For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives.

The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not too large (32 bytes) and the odds of a false positive/collision are low enough that the world will end before you ever encounter one. SHA256 is the de-facto standard for this kind of thing and I'd be very surprised if he'd done anything else.

Re: Hyperspace

#159

Post by [deleted] »

theamk wrote: post #143
strunz wrote: post #123

Earlier quoted context omitted.

I wonder how any comments about hard links will be in these comments by people misunderstanding what this app does.

if file is not going to be modified (in the low-level sense - open("w") on the filename; as opposed to rename-and-create-new), then reflinks (what this app does) and hardlinks act somewhat identically. For example if you have multiple node_modules, or app installs, or source photos/videos (ones you don't edit), or music archives, then hardlinks work just fine.

[deleted]

Re: Hyperspace

#160

Post by rappatic »

amzin wrote: post #146
taneliv wrote: post #88

Earlier quoted context omitted.

On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot d…

Is there a FS that keeps only diffs in clone files? It would be neat

I wondered that too.

If we only have two files, A and its duplicate B with some changes as a diff, this works pretty well. Even if the user deletes A, the OS could just apply the diff to the file on disk, unlink A, and assign B to that file.

But if we have A and two different diffs B1 and B2, then try to delete A, it gets a little murkier. Either you do the above process and recalculate the diff for B2 to make it a diff of B1; or you keep the original A floating around on disk, not linked to any file.

Similarly, if you try to modify A, you'd need to recalculate the diffs for all the duplicates. Alternatively, you could do version tracking and have the duplicate's diffs be on a specific version of A. Then every file would have a chain of diffs stretching back to the original content of the file. Complex but could be useful.

It's certainly an interesting concept but might be more trouble than it's worth.

Post reply on HN