Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperspace

hypercritical.co

Post reply on HN
Print view View original

351–360 of 503 posts

Re: Hyperspace

#351

Post by jonhohle »

MBCook wrote: post #180
borland wrote: post #158

Earlier quoted context omitted.

I don't know exactly what Siracusa is doing here, but I can take an educated guess: For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives. The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not t…

You can start with the size, which is probably really unique. That would likely cut down the search space fast. At that point maybe it’s better to just compare byte by byte? You’ll have to read the whole file to generate the hash and if you just compare the bytes there is no chance of hash collision no matter how small. Plus if you find a difference in bytes 1290 you can just stop there instead of reading the whole t…

To make dedup[0] fast, I use a tree with device id, size, first byte, last byte, and finally SHA-256. Each of those is only used if there is a collision to avoid as many reads as possible. dedup doesn’t do a full file compare, because if you’ve found a file with the same size, first and last bytes, and SHA-256 you’ve also probably won the lottery several times over and can afford data recovery.

This is the default for ZFS deduplication and git does something similar with size and far weaker SHA-1. I would add a test for SHA-256 collisions, but no one has seemed to find a working example yet.

0 - https://github.com/ttkb-oss/dedup

Re: Hyperspace

#352

Post by scarface_74 »

dewey wrote: post #224
diggan wrote: post #75

> Like all my apps, Hyperspace is a bit difficult to explain. I’ve attempted to do so, at length, in the Hyperspace documentation. I hope it makes enough sense to enough people that it will be a useful addition to the Mac ecosystem. Am I missing something, or isn't it a "file de-duplicator" with a nice UI/UX? Sounds pretty simple to describe, and tells you why it's useful with just two words.

The author of the software is a file system enthusiast (so much that in the podcast he's a part of they have a dedicated sound effect every time "filesystem" comes up), a long time blogger and macOS reviewer. So you'll have to see it in that context while documenting every bit and the technical details behind it is important to him...even if it's longer than a tag line on a landing page. In times where documentation…

Before that he was known for his exhaustive reviews of OS X on Ars Technica

https://arstechnica.com/author/john-siracusa/

Re: Hyperspace

#353

Post by saagarjha »

pmarreck wrote: post #242
DontBreakAlex wrote: post #181

Nice, but I'm not getting a subscription for a filesystem utility. Had it been a one-time $5 license, I would have bought it. At the current price, it's literally cheaper to put files in a S3 bucket or outright buy an SSD.

Claude 3.7 just rewrote the whole thing (just based on reading the webpage description) as a commandline app for me, so there's that. And because it has no Internet access yet (and because I prompted it to use a workaround like this in that circumstance), the first thing it asked me to do (after hallucinating the functionality first, and then catching itself) was run `curl https://hypercritical.co/hyperspace/ | sed '…

Would you trust Claude with your hard drive?

Re: Hyperspace

#354

Post by jonhohle »

karparov wrote: post #307
borland wrote: post #158

Earlier quoted context omitted.

I don't know exactly what Siracusa is doing here, but I can take an educated guess: For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives. The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not t…

This can be done much faster and safer. You can group all files into buckets, and as soon as a bucket is empty, discard it. If in the end there are still files in the same bucket, they are duplicates. Initially all files are in the same bucket. You now iterate over differentiators which given two files tell you whether they are maybe equal or definitely not equal. They become more and more costly but also more and mo…

Not surprisingly, differentiator 2 can just be the first byte (or machine word). Differentiator 3 can be the last byte (or word). At that point, 99.99% (in practice more 9s) of files are different and you’re read at most 2 blocks per file. I haven’t figured out a good differentiator 3 prior to hashing, but it’s already so rare, that it’s not worth it, in my experience.

Re: Hyperspace

#355

Post by throwaway2037 »

taneliv wrote: post #88
bob1029 wrote: post #48

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safel…

On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot d…

From a file system design perspective, does anyone know why ZFS chose to use block clones, instead of file clones?

Re: Hyperspace

#356

Post by wizzwizz4 »

pmarreck wrote: post #242
DontBreakAlex wrote: post #181

Nice, but I'm not getting a subscription for a filesystem utility. Had it been a one-time $5 license, I would have bought it. At the current price, it's literally cheaper to put files in a S3 bucket or outright buy an SSD.

Claude 3.7 just rewrote the whole thing (just based on reading the webpage description) as a commandline app for me, so there's that. And because it has no Internet access yet (and because I prompted it to use a workaround like this in that circumstance), the first thing it asked me to do (after hallucinating the functionality first, and then catching itself) was run `curl https://hypercritical.co/hyperspace/ | sed '…

Ooh, could you share the source code? That seems like a perfect example for my "relying on AI code generation will subtly destroy your data" presentation.

Re: Hyperspace

#359

Post by pfranz »

gurjeet wrote: post #331
andrewla wrote: post #106

Earlier quoted context omitted.

This does not use hard links or symlinks; this uses a feature of the filesystem that allows the creation of copy-on-write clones. [1] [1] https://en.wikipedia.org/wiki/Apple_File_System#Clones

So albertzeyer's script can be adapted to use `cp -c` command, to achieve the same effect as Hyperspace.

If you'd like. In the blog post he says he wrote the prototype in an afternoon. Hyperspace does try hard to preserve unique metadata as well as other protections.
Post reply on HN