Nice, but I'm not getting a subscription for a filesystem utility. Had it been a one-time $5 license, I would have bought it. At the current price, it's literally cheaper to put files in a S3 bucket or outright buy an SSD.
Hyperspace
181–190 of 503 posts
Re: Hyperspace
#182Earlier quoted context omitted.
He wanted to write it in Swift 6. Does it support older OS versions?
Swift 6 is not the problem. It's backward compatible. The problem is SwiftUI. It's very new, still barely usable on the Mac, but they are adding lots of new features every macOS release. If you want to support older versions of macOS you can't use the nice stuff they just released. Eg. pointerStyle() is a brand new macOS 15 API that is very useful.
I can’t remember for sure but there may also have been a recent file system API he said he needed. Or a bug that he had to wait for a fix on.
Re: Hyperspace
#183Earlier quoted context omitted.
If it saved 8.1GB, by your measure it'd also not be a big win?
This is basically only a win on macOS, and only because Apple charges through the nose for disk space. Ex - On my non-apple machines, 8GB is trivial. I load them up with the astoundingly cheap NVMe drives in the multiple terabyte range (2TB for ~$100, 4TB for ~$250) and I have a cheap NAS. So that "big win" is roughly 40 cents of hardware costs on the direct laptop hardware. Hardly worth the time and effort involved,…
> This is basically only a win on macOS, and only because Apple charges through the nose for disk space
You do realize that this software is only available on macOS, and only works because of Apple's APFS filesystem? You're essentially complaining that medicine is only a win for people who are sick.
Re: Hyperspace
#184Earlier quoted context omitted.
What kind of changes could you make to one clone that would still qualify it as a clone? If there are changes, it's no longer the same file. Even after reading the How It Works[0] link, I'm not groking how it works. Is it making some sort of delta/diff that is applied to the original file? That's not possible for every file format like large media files. I could see that being interesting for text based files, but th…
If I understand correctly, a COW clone references the same contents (just like a hardlink) as long as all the filesystem references are pointing to identical file contents. Once you open one of the reference handles and modify the contents, the copy-on-write process is invoked by the filesystem, and the underlying data is copied into a new, separate file with your new changes, breaking the link. Comparing with a hard…
ah, that's where the copy-on-write takes place. sometimes, just reading it written by someone else is the knock upside the head I need.
Re: Hyperspace
#185What algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?
I don't know exactly what Siracusa is doing here, but I can take an educated guess: For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives. The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not t…
I think the prob. is not so low. I remember reading here about a person getting a foto of another chat in a chat application, which was using sha in the background. I do not recall all the details, it is improbable, but possible.
Re: Hyperspace
#186What algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?
I don't know exactly what Siracusa is doing here, but I can take an educated guess: For each candidate file, you need some "key" that you can use to check if another candidate file is the same. There can be millions of files so the key needs to be small and quick to generate, but at the same time we don't want any false positives. The obvious answer today is a SHA256 hash of the file's contents; It's very fast, not t…
[deleted]
Re: Hyperspace
#187Earlier quoted context omitted.
Even if I have both a Mac and iPhone, but happen to use my Linux computer right now, it seems like the store page ( https://apps.apple.com/us/app/hyperspace-reclaim-disk-space/... ) is not showing the price, probably because I'm not actively on a Apple device? Seems like a poor UX even for us Mac users.
It's buried under a drop-down in the "Information" section, under "In-App Purchases". I agree, it's not the greatest.
It’s a side effect of the terrible store design.
It’s a free app because you don’t have to buy it to run it. It will tell you how much space it can save you for free. So you don’t have to waste $20 to find out it only would’ve been 2kb.
But that means the parts you actually have to buy are in app purchases, which are always hidden on the store pages.
Re: Hyperspace
#188Its interesting how Linux tools are all free when even trivial mac tools are being sold. Nothing against someone trying to monetize but the linux culture sure is nice!
Re: Hyperspace
#189> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safel…
Windows Server does this for NTFS and ReFS volumes. I used it quite a bit on ReFS w/ Hyper-V VMs and it worked wonders . Cut my storage usage down by ~45% with a majority of Windows Server VMs running a mix of 2016/2019 at the time.
Yep. At a previous job we had a file server that we published Windows build output to.
There were about 1000 copies of the same pre-requisite .NET and VC++ runtimes (each build had one) and we only paid for the cost of storing it once. It was great.
It is worth pointing out though, that on Windows Server this deduplication is a background process; When new duplicate files are created, they genuinely are duplicates and take up extra space, but once in a while the background process comes along and "reclaims" them, much like the Hyperspace app here does.
Because of this (the background sweep process is expensive), it doesn't run all the time and you have to tell it which directories to scan.
If you want "real" de-duplication, where a duplicate file will never get written in the first place, then you need something like ZFS
Re: Hyperspace
#190Earlier quoted context omitted.
On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot d…
Files are always represented as lists of blocks or block spans within a file system. Individual blocks could in theory be partially shared between files at the complexity cost of a reference counter for each block. So changing a single byte in a copy on write file could take the same time regardless of file size because only the affected bock would have to be duplicated. I don't know at all how MacOS implements this…
APFS is a copy on write filesystem if you use the right APIs, so it does what you describe but only for entire files.
I believe as soon as you change a single bite you get a complete copy that’s your own.
And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.