Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rename files to match hash of contents

gist.github.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: Rename files to match hash of contents

#21

Post by kardos »

spangry wrote: post #20
kardos wrote: post #16

Earlier quoted context omitted.

If you had a copy of the two image files from my second link, this 'dupe detector' would erroneously flag one as a dupe. Also, what of truncating the hashes? I don't get why people try to justify using severely weakened things when using the non-broken (ie, secure) version is a /trivial/ drop in replacement...

I'm not trying to justify anything. I'm just trying to suggest you're labouring under a misapprehension. And this has nothing to do with security. I'm guessing you've heard the (good) advice that md5 is not a secure hashing function for, say, storing passwords, and then promptly joined the 'md5 is bad for all the things' cargo cult. So while you're correct about the two images on that blog, the only reason why you'd…

fdupes is not a problem assuming wikipedia's description [1] is correct: "It first compares file sizes, partial MD5 signatures, full MD5 signatures, and then performs a byte-by-byte comparison for verification."

I was unimpressed by the md5 used in the shell script at the original link, which is using a truncated md5...

[1] https://en.wikipedia.org/wiki/Fdupes

Re: Rename files to match hash of contents

#23

Post by spangry »

kardos wrote: post #21
spangry wrote: post #20

Earlier quoted context omitted.

I'm not trying to justify anything. I'm just trying to suggest you're labouring under a misapprehension. And this has nothing to do with security. I'm guessing you've heard the (good) advice that md5 is not a secure hashing function for, say, storing passwords, and then promptly joined the 'md5 is bad for all the things' cargo cult. So while you're correct about the two images on that blog, the only reason why you'd…

fdupes is not a problem assuming wikipedia's description [1] is correct: "It first compares file sizes, partial MD5 signatures, full MD5 signatures, and then performs a byte-by-byte comparison for verification." I was unimpressed by the md5 used in the shell script at the original link, which is using a truncated md5... [1] https://en.wikipedia.org/wiki/Fdupes

Ok, fair enough. I would agree with the view that using md5, presumably for the faster performance, is probably not the best trade-off to be making here. Unless we're dealing with an NVMe drive (or something more exotic), you're likely to be IO bound even if using more computationally intensive hashing functions.

And if you are deduping on really fast storage, you'd get way better performance (with comparable safety) using something like xxHash64 (https://cyan4973.github.io/xxHash/).

Post reply on HN