I made a command line utility called `dedup` a while back to do the same thing. It has a dry-run mode, will “intelligently” choose the best clone source, understands hard links and other clones, preserves metadata, deals with HFS compressed files properly. It hasn’t destroyed any of my own data, but like any file system tool, use at your own risk.
Hyperspace
341–350 of 503 posts
Re: Hyperspace
#342Earlier quoted context omitted.
From the FAQ: > If some eligible files were found, the amount of disk space that can be reclaimed is shown next to the “Potential Savings” label. To proceed any further, you will have to make a purchase. Once the app’s full functionality is unlocked, a “Review Files” button will become available after a successful scan. This will open the Review Window. I half remember this being discussed on ATP; the logic being tha…
> the logic being that if you have the list of files, you will just go and de-dupe them yourself. If you can do that, you can check for duplicates yourself anyway. It's not like there aren't already dozens of great apps that dedupe.
> there aren't already dozens of great apps that dedupe.
Most of those delete rather than use the features of APFS.
Re: Hyperspace
#343What jumped out to me: > Finally, at WWDC 2017, Apple announced Apple File System (APFS) for macOS (after secretly test-converting everyone’s iPhones to APFS and then reverting them back to HFS+ as part of an earlier iOS 10.x update in one of the most audacious technological gambits in history). How can you revert a FS change like that if it goes south? You'd certainly exercise the code well but also it seems like yo…
I watched the section from the talk [0] and there's no details given really, other than that it was done as a test of consistency. I've blown so many things up in production that I'm not sure if I could every pull the trigger on such a large migration
Re: Hyperspace
#344Earlier quoted context omitted.
Even using sha-256 or greater type of hashing, I'd still have concerns about letting a system make deletion decisions without my involvement. I've even been part of de-dupe efforts, so maybe my hesitation is just because I wrote some of the code and I know I'm not perfect in my coding or even my algo decision trees. I know that any mistake I made would not be of malice but just ignorance or other stupid mistake. I've…
Now you understand why this app costs more than 2x the price of alternatives such as diskDedupe. Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no. The author - John Siracusa - has been a profes…
Best course of action is to not trust John, and just wait for a year of the app out the wild, until everyone else trusts John . I have enough hard drive space in the meantime to not rush into trusting John.
Re: Hyperspace
#345Downloaded. Ran it. Tells me "900" files can be cleaned. No summary, no list. But I was at least asked to buy the app. Why would I buy the app if I have no idea if it'll help?
If you don’t mind CLI tools, You can try dedup - https://github.com/ttkb-oss/dedup . Use the —-dry-run option to get a list of files that would be merged without modifying anything and how much space would be saved.
Re: Hyperspace
#346Earlier quoted context omitted.
On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot d…
Is there a FS that keeps only diffs in clone files? It would be neat
APFS shares blocks so only blocks that changed are no longer shared. Since a block is the smallest atomic unit (except maybe an inode) in a FS, that’s the best level of granularity to expect.
Re: Hyperspace
#347Its interesting how Linux tools are all free when even trivial mac tools are being sold. Nothing against someone trying to monetize but the linux culture sure is nice!
It's not that nice to call someone's work they spent months on "trivial" without knowing anything about the internals and what they ran into.
I didnt mean that the work was trivial, but the tool seems like something that could be a simple CLI
Re: Hyperspace
#348Downloaded. Ran it. Tells me "900" files can be cleaned. No summary, no list. But I was at least asked to buy the app. Why would I buy the app if I have no idea if it'll help?
This reminds me - Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs. They all inevitably found at least few KB to be reclaimed through their magic even if the same optimizer was run back to back with itself and allowed to "optimize" things. That is, on each run they always find extra memory to be freed. They ultimately did nothing but claim they did the wo…
>Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs
You may find this interesting: Investigations into SoftRAM 95 by Raymond Chen [1] and Mark Russinovich [2] respectively.
"They implemented only one compression algorithm.
It was memcpy."
[1] https://devblogs.microsoft.com/oldnewthing/20211111-00/?p=10...
[2] https://www.drdobbs.com/parallel/inside-softram-95/184409937
Re: Hyperspace
#349I made a command line utility called `dedup` a while back to do the same thing. It has a dry-run mode, will “intelligently” choose the best clone source, understands hard links and other clones, preserves metadata, deals with HFS compressed files properly. It hasn’t destroyed any of my own data, but like any file system tool, use at your own risk. 0 - https://github.com/ttkb-oss/dedup
Just tried it, and it works well! I didn't realize the potential of this technique until I saw just how many dupes there were of certain types of files, especially in node_modules. It wasn't uncommon to see it replace 50 copies of some js file with one, and that was just in a specific subdirectory.
I see it is "pre-release" and sort of low GH stars (== usage?), so I'm curious about the stability since this type of tool is relatively scary if buggy.
Re: Hyperspace
#350Downloaded. Ran it. Tells me "900" files can be cleaned. No summary, no list. But I was at least asked to buy the app. Why would I buy the app if I have no idea if it'll help?
This reminds me - Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs. They all inevitably found at least few KB to be reclaimed through their magic even if the same optimizer was run back to back with itself and allowed to "optimize" things. That is, on each run they always find extra memory to be freed. They ultimately did nothing but claim they did the wo…
> Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs. They ultimately did nothing but claim they did the work. Must've sold pretty well nonetheless.
QEMM worked by remapping stuff into extended memory - in a time that most software wasn't interested in using it. It worked as advertised.
Quarterdeck made good stuff all around. Desq and DesqView/X were amazing multitaskers. Way snappier than Windows and ran on little to nothing.