Rename files to match hash of contents
11–20 of 25 posts
Re: Rename files to match hash of contents
#12Earlier quoted context omitted.
This is also how FreeBSD Update and Portsnap store files. This technique has been around for a long time.
https://en.wikipedia.org/wiki/Content-addressable_storage
Right. I'm not a fan of the terminology though; I prefer hash-addressed storage to avoid the potential confusion with associative memory.
Re: Rename files to match hash of contents
#13I wrote something similar once, but only for gifs and it also fixes file extensions for a few mimetypes.
Re: Rename files to match hash of contents
#14Nirsofts 'hashmyfiles' has this functionality built in already, known as duplicate search mode. Works extremely well. http://www.nirsoft.net/utils/hash_my_files.html
Re: Rename files to match hash of contents
#15It would be nice to turn this into a program that stores the previous name so they can be renamed back after deduplicating. Very cool!
Why rename them at all? There are already good tools for duplicate detection. An example is fdupes [1], which is smart enough to rule out dupes by other tricks like partial hashes etc., so you can avoid hashing some of the files. [1] https://github.com/adrianlopezroche/fdupes Edit: just noticed that it's using md5, which is broken [2], and that it's using truncated md5 hashes.....! [2] https://natmchugh.blogspot.ca/2…
Using md5 is only a problem here if someone has actually gained access to your files and then gone to the trouble of secretly adding new files and calculating/brute-forcing the correct 'chosen-prefixes' to ensure a clash. It would be a pretty weird attack to mount, that's for sure.
md5 is fine for deduplicating. It's extremely improbable you'd 'organically' get a md5 hash clash for two different files.
Re: Rename files to match hash of contents
#16Earlier quoted context omitted.
Why rename them at all? There are already good tools for duplicate detection. An example is fdupes [1], which is smart enough to rule out dupes by other tricks like partial hashes etc., so you can avoid hashing some of the files. [1] https://github.com/adrianlopezroche/fdupes Edit: just noticed that it's using md5, which is broken [2], and that it's using truncated md5 hashes.....! [2] https://natmchugh.blogspot.ca/2…
Using md5 is only a problem here if someone has actually gained access to your files and then gone to the trouble of secretly adding new files and calculating/brute-forcing the correct 'chosen-prefixes' to ensure a clash. It would be a pretty weird attack to mount, that's for sure. md5 is fine for deduplicating. It's extremely improbable you'd 'organically' get a md5 hash clash for two different files.
If you had a copy of the two image files from my second link, this 'dupe detector' would erroneously flag one as a dupe.
Also, what of truncating the hashes?
I don't get why people try to justify using severely weakened things when using the non-broken (ie, secure) version is a /trivial/ drop in replacement...
Re: Rename files to match hash of contents
#17It would be nice to turn this into a program that stores the previous name so they can be renamed back after deduplicating. Very cool!
Or just use git-annex
Re: Rename files to match hash of contents
#18Earlier quoted context omitted.
Why rename them at all? There are already good tools for duplicate detection. An example is fdupes [1], which is smart enough to rule out dupes by other tricks like partial hashes etc., so you can avoid hashing some of the files. [1] https://github.com/adrianlopezroche/fdupes Edit: just noticed that it's using md5, which is broken [2], and that it's using truncated md5 hashes.....! [2] https://natmchugh.blogspot.ca/2…
Using md5 is only a problem here if someone has actually gained access to your files and then gone to the trouble of secretly adding new files and calculating/brute-forcing the correct 'chosen-prefixes' to ensure a clash. It would be a pretty weird attack to mount, that's for sure. md5 is fine for deduplicating. It's extremely improbable you'd 'organically' get a md5 hash clash for two different files.
[deleted]
Re: Rename files to match hash of contents
#19Earlier quoted context omitted.
Using md5 is only a problem here if someone has actually gained access to your files and then gone to the trouble of secretly adding new files and calculating/brute-forcing the correct 'chosen-prefixes' to ensure a clash. It would be a pretty weird attack to mount, that's for sure. md5 is fine for deduplicating. It's extremely improbable you'd 'organically' get a md5 hash clash for two different files.
[deleted]
[deleted]
Re: Rename files to match hash of contents
#20Earlier quoted context omitted.
Using md5 is only a problem here if someone has actually gained access to your files and then gone to the trouble of secretly adding new files and calculating/brute-forcing the correct 'chosen-prefixes' to ensure a clash. It would be a pretty weird attack to mount, that's for sure. md5 is fine for deduplicating. It's extremely improbable you'd 'organically' get a md5 hash clash for two different files.
If you had a copy of the two image files from my second link, this 'dupe detector' would erroneously flag one as a dupe. Also, what of truncating the hashes? I don't get why people try to justify using severely weakened things when using the non-broken (ie, secure) version is a /trivial/ drop in replacement...
I'm not trying to justify anything. I'm just trying to suggest you're labouring under a misapprehension. And this has nothing to do with security. I'm guessing you've heard the (good) advice that md5 is not a secure hashing function for, say, storing passwords, and then promptly joined the 'md5 is bad for all the things' cargo cult.
So while you're correct about the two images on that blog, the only reason why you'd get a clash is because the author of that blog post spent ~15 hours on an AWS GPU instance to generate the correct prefixes which, when appended to those files, results in a clash.
So, I guess if you are in the habit of grabbing random files from your hdd, loading them on to an AWS GPU instance for 15 hours (per file) and generating hash collisions, then yeah, don't use fdupes.