Rename files to match hash of contents

#1

Post by SimplGy » Sun, Nov 27, 2016, 12:39 AM UTC

Rename files to match hash of contents
gist.github.com

Re: Rename files to match hash of contents

#2

Post by tucaz » Sun, Nov 27, 2016, 2:29 AM UTC

It would be nice to turn this into a program that stores the previous name so they can be renamed back after deduplicating.

Very cool!

Re: Rename files to match hash of contents

#3

Post by d4l3k » Sun, Nov 27, 2016, 2:54 AM UTC

tucaz wrote: post #2

It would be nice to turn this into a program that stores the previous name so they can be renamed back after deduplicating. Very cool!

Why not just have a program that iterates through all of the files, hashes them, stores them in a map/dict and then reports if there's a duplicate? Seems easier than renaming everything multiple times.

Re: Rename files to match hash of contents

#4

Post by sgentle » Sun, Nov 27, 2016, 2:56 AM UTC

Fun! This is similar to how git stores files internally. You can do some neat tricks like this:

  $ ls
  01.jpg      03.jpg      03_copy.jpg 04.jpg      05.jpg

  $ git init
  Initialized empty Git repository in /tmp/test/.git/

  $ git hash-object -w *
  82f7d50fc89d2fd47150aff539ea4acf45ec1589
  0080672bc4f248c400d569cce1a2a3d743eb1331
  0080672bc4f248c400d569cce1a2a3d743eb1331
  58db57b10c219b9b71f0223e58a6dc0d51cfe207
  05dcde743807bddaf55ad1231572c1365d4db4af

  $ find .git/objects -type f
  .git/objects/00/80672bc4f248c400d569cce1a2a3d743eb1331
  .git/objects/05/dcde743807bddaf55ad1231572c1365d4db4af
  .git/objects/58/db57b10c219b9b71f0223e58a6dc0d51cfe207
  .git/objects/82/f7d50fc89d2fd47150aff539ea4acf45ec1589

If you're curious, you can read more about how it works here: https://git-scm.com/book/en/v1/Git-Internals-Git-Objects

Re: Rename files to match hash of contents

#5

Post by kardos » Sun, Nov 27, 2016, 2:57 AM UTC

tucaz wrote: post #2

It would be nice to turn this into a program that stores the previous name so they can be renamed back after deduplicating. Very cool!

Why rename them at all? There are already good tools for duplicate detection. An example is fdupes [1], which is smart enough to rule out dupes by other tricks like partial hashes etc., so you can avoid hashing some of the files.

[1] https://github.com/adrianlopezroche/fdupes

Edit: just noticed that it's using md5, which is broken [2], and that it's using truncated md5 hashes.....!

[2] https://natmchugh.blogspot.ca/2015/02/create-your-own-md5-co...

Re: Rename files to match hash of contents

#6

Post by cperciva » Sun, Nov 27, 2016, 3:03 AM UTC

sgentle wrote: post #4

Fun! This is similar to how git stores files internally. You can do some neat tricks like this: $ ls 01.jpg 03.jpg 03_copy.jpg 04.jpg 05.jpg $ git init Initialized empty Git repository in /tmp/test/.git/ $ git hash-object -w * 82f7d50fc89d2fd47150aff539ea4acf45ec1589 0080672bc4f248c400d569cce1a2a3d743eb1331 0080672bc4f248c400d569cce1a2a3d743eb1331 58db57b10c219b9b71f0223e58a6dc0d51cfe207 05dcde743807bddaf55ad1231572c…

This is also how FreeBSD Update and Portsnap store files. This technique has been around for a long time.

Re: Rename files to match hash of contents

#7

Post by sliken » Sun, Nov 27, 2016, 3:46 AM UTC

Be warned that this (by default) only looks at part of the file. Seems like a poor default.

Re: Rename files to match hash of contents

#8

Post by sliken » Sun, Nov 27, 2016, 4:12 AM UTC

d4l3k wrote: post #3

tucaz wrote: post #2

It would be nice to turn this into a program that stores the previous name so they can be renamed back after deduplicating. Very cool!

Why not just have a program that iterates through all of the files, hashes them, stores them in a map/dict and then reports if there's a duplicate? Seems easier than renaming everything multiple times.

That's basically fdupe. Also you only have to hash files with the same length, if they aren't the same length you can be quite sure they aren't the same file.

Even such a simple optimization can make a huge difference on a large directory of images or MP3s.

Re: Rename files to match hash of contents

#9

Post by askvictor » Sun, Nov 27, 2016, 4:27 AM UTC

Don't modern filesystems allow you to store metadata like this separately to the filename or file data?

Re: Rename files to match hash of contents

#10

Post by patrickmn » Sun, Nov 27, 2016, 5:30 AM UTC

cperciva wrote: post #6

sgentle wrote: post #4

Fun! This is similar to how git stores files internally. You can do some neat tricks like this: $ ls 01.jpg 03.jpg 03_copy.jpg 04.jpg 05.jpg $ git init Initialized empty Git repository in /tmp/test/.git/ $ git hash-object -w * 82f7d50fc89d2fd47150aff539ea4acf45ec1589 0080672bc4f248c400d569cce1a2a3d743eb1331 0080672bc4f248c400d569cce1a2a3d743eb1331 58db57b10c219b9b71f0223e58a6dc0d51cfe207 05dcde743807bddaf55ad1231572c…

This is also how FreeBSD Update and Portsnap store files. This technique has been around for a long time.

https://en.wikipedia.org/wiki/Content-addressable_storage