391–400 of 503 posts
Earlier quoted context omitted.
It didn't tell you how much disk space? It's supposed to. It only told you the number of files?
Only told me the number of files and it didn't even provide a list of files (their paths, etc.)
Earlier quoted context omitted.
You can start with the size, which is probably really unique. That would likely cut down the search space fast. At that point maybe it’s better to just compare byte by byte? You’ll have to read the whole file to generate the hash and if you just compare the bytes there is no chance of hash collision no matter how small. Plus if you find a difference in bytes 1290 you can just stop there instead of reading the whole t…
To make dedup[0] fast, I use a tree with device id, size, first byte, last byte, and finally SHA-256. Each of those is only used if there is a collision to avoid as many reads as possible. dedup doesn’t do a full file compare, because if you’ve found a file with the same size, first and last bytes, and SHA-256 you’ve also probably won the lottery several times over and can afford data recovery. This is the default fo…
For those mentioning that there's no price listed, it's not that easy as in the App Store the price varies by country. You can open the App Store link and then look at "In App Purchases" though. For me on the German store it looks like this: Unlock for One Year 22,99 € Unlock for One Month 9,99 € Lifetime Unlock 59,99 € So it supports both one time purchases and subscriptions. Depending on what you prefer. More about…
Its interesting how Linux tools are all free when even trivial mac tools are being sold. Nothing against someone trying to monetize but the linux culture sure is nice!
I made a command line utility called `dedup` a while back to do the same thing. It has a dry-run mode, will “intelligently” choose the best clone source, understands hard links and other clones, preserves metadata, deals with HFS compressed files properly. It hasn’t destroyed any of my own data, but like any file system tool, use at your own risk. 0 - https://github.com/ttkb-oss/dedup
Replying to myself now that I've had a chance to try the scan, but not the deduplication. I work with disc images, program binaries, intermediate representations in a workspace that's 7.6G. A few notes: * By default it doesn't scan everything. It ignores all files but those in an allow list. The way the allow list is structured, it seems like Hyperspace needs to understand the content of a file. As an end user, I hav…
With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..
Earlier quoted context omitted.
Yes true, but that page also covers some recent improvements to de-duplication that might assist.
Really? I haven't looked at this ZFS feature in a few years so I will take a look EDIT: Is this referring to the "fast" dedup feature?
Earlier quoted context omitted.
In regards to the second point, this isn't correct for ZFS: "If several files contain the same pieces (blocks) of data or any other pool data occurs more than once in the pool, ZFS stores just one copy of it. Instead of storing many copies of a book it stores one copy and an arbitrary number of pointers to that one copy." [0]. So changing one byte of a large file will not suddenly result in writing the whole file to…
Not the whole file but it would duplicate the block. GP didn't claim that the whole file is copied.
Earlier quoted context omitted.
On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot d…
In regards to the second point, this isn't correct for ZFS: "If several files contain the same pieces (blocks) of data or any other pool data occurs more than once in the pool, ZFS stores just one copy of it. Instead of storing many copies of a book it stores one copy and an arbitrary number of pointers to that one copy." [0]. So changing one byte of a large file will not suddenly result in writing the whole file to…
Earlier quoted context omitted.
> Which means if you actually edited those files, you might fill up your HD much more quickly than you expected. I'm not sure if this is what you intended, but just to be sure: writing changes to a cloned file doesn't immediately duplicate the entire file again in order to write those changes — they're actually written out-of-line, and the identical blocks are only stored once. From [the docs](^1) posted in a sibling…
The key is “unmodified” and how APFS knows or doesn’t know whether they are modified. How many apps write on block boundaries or even mutate just in disk data that has changed vs overwriting or replacing atomically? For most applications there is no benefit and a significant risk of corruption. So APFS supports it, but there is no way to control what an app is going to do, and after it’s done it, no way to know what…
I don't understand what makes you think there's a significant risk of corruption. Are you talking about the risk of something modifying a file while the dedupe is happening? Or do you think there's risk associated with just having deduplicated files on disk?
Earlier quoted context omitted.
No because it isn't getting rid of the duplicate, it's using a feature of APFS that allows for duplicates to exist separately but share the same internal data.
It does get rid of the duplicate. The duplicate data is deleted and a hard link is created in its place.