Live data from Hacker News

Hyperspace

hypercritical.co

101–110 of 503 posts

Re: Hyperspace

#101

Earlier quoted context omitted.

No because it isn't getting rid of the duplicate, it's using a feature of APFS that allows for duplicates to exist separately but share the same internal data.

It does get rid of the duplicate. The duplicate data is deleted and a hard link is created in its place.

It does not make hard links. It makes copy-on-write clones.

Re: Hyperspace

#102

Earlier quoted context omitted.

No because it isn't getting rid of the duplicate, it's using a feature of APFS that allows for duplicates to exist separately but share the same internal data.

It does get rid of the duplicate. The duplicate data is deleted and a hard link is created in its place.

No, because it's not actually a hard link -- if you modify one of the files they'll diverge.

Re: Hyperspace

#103

$ rmlint -c sh:link -L -y s -p -T duplicates will produce a script which, if run, will hardlink duplicates

That's not what this app is doing though. APFS clones are copy-on-write pointers to the same data, not hardlinks.

If you replace `sh:link` with `sh:clone` instead, it will.

> clone: reflink-capable filesystems only. Try to clone both files with the FIDEDUPERANGE ioctl(3p) (or BTRFS_IOC_FILE_EXTENT_SAME on older kernels). This will free up duplicate extents while preserving the metadata of both. Needs at least kernel 4.2.

Re: Hyperspace

#104

Earlier quoted context omitted.

[I was wrong, see below.—cw] It doesn't do the same thing. An APFS clone/copy-on-write clone is not the same as a hard or soft link. https://eclecticlight.co/2019/01/05/aliases-hard-links-symli...

Your source points out that: You can also create [APFS (copy on write) clones] in Terminal using the command `cp -c oldfilename newfilename` where the c option requires cloning rather than a regular copy. `fclones dedupe` uses the same command[1]: if cfg!(target_os = "macos") { result.push(format!("cp -c {target} {link}")); [1] https://github.com/pkolaczk/fclones/blob/555cde08fde4e700b25...

I stand corrected, thank you!

Re: Hyperspace

#105
post #75

> Like all my apps, Hyperspace is a bit difficult to explain. I’ve attempted to do so, at length, in the Hyperspace documentation. I hope it makes enough sense to enough people that it will be a useful addition to the Mac ecosystem. Am I missing something, or isn't it a "file de-duplicator" with a nice UI/UX? Sounds pretty simple to describe, and tells you why it's useful with just two words.

[deleted]

Re: Hyperspace

#106

I wrote a similar (but simpler) script which would replace a file by a hardlink if it has the same content. My main motivation was for the packages of Python virtual envs, where I often have similar packages installed, and even if versions are different, many files would still match. Some of the packages are quite huge, e.g. Numpy, PyTorch, TensorFlow, etc. I got quite some disk space savings from this. https://githu…

This does not use hard links or symlinks; this uses a feature of the filesystem that allows the creation of copy-on-write clones. [1]

[1] https://en.wikipedia.org/wiki/Apple_File_System#Clones

Re: Hyperspace

#108
post #66

Earlier quoted context omitted.

The relevant number (missing from above) is the total amount of space on that storage device. If it saves 1GB on a 8TB drive, it's not a big win.

If it saved 8.1GB, by your measure it'd also not be a big win?

This is basically only a win on macOS, and only because Apple charges through the nose for disk space.

Ex - On my non-apple machines, 8GB is trivial. I load them up with the astoundingly cheap NVMe drives in the multiple terabyte range (2TB for ~$100, 4TB for ~$250) and I have a cheap NAS.

So that "big win" is roughly 40 cents of hardware costs on the direct laptop hardware. Hardly worth the time and effort involved, even if the risk is zero (and I don't trust it to be zero).

If it's just "storage" and I don't need it fast (the perfect case for this type of optimization) I throw it on my NAS where it's cheaper still... Ex - it's not 40 cents saved, it's ~10.

---

At least for me, 8GB is no longer much of a win. It's a rounding error on the last LLM model I downloaded.

And I'd suggest that basically anyone who has the ability to not buy extortionately priced drives soldered onto a mainboard is not really winning much here either.

I picked up a quarter off the ground on my walk last night. That's a bigger win.

Re: Hyperspace

#109

Earlier quoted context omitted.

No because it isn't getting rid of the duplicate, it's using a feature of APFS that allows for duplicates to exist separately but share the same internal data.

Is it not the same as a hard link (which I believe are supported on Mac too)?

With a hard link, the content of each of the two 'files' are identical in perpetuity.

With APFS Clones, the contents start off identical, but can be changed independently. If you change a small part of a file, those block(s) will need to be created, but the existing blocks will continue to be shared with the clone.

Re: Hyperspace

#110
post #75

> Like all my apps, Hyperspace is a bit difficult to explain. I’ve attempted to do so, at length, in the Hyperspace documentation. I hope it makes enough sense to enough people that it will be a useful addition to the Mac ecosystem. Am I missing something, or isn't it a "file de-duplicator" with a nice UI/UX? Sounds pretty simple to describe, and tells you why it's useful with just two words.

[deleted]
Post reply on HN