Detecting duplicate images with Python

rplnt wrote: post #2

I feel like this is doing something that was already done better (probably), but I enjoyed the article anyways. Nicely summarized what, why, and how.

Any idea of what might be a better approach?

Re: Detecting duplicate images with Python

#5

Post by ambirex » Thu, Apr 03, 2014, 11:56 AM UTC

Several years ago I tried matching images on histogram comparisons, I realized it wasn't going to be a perfect match, but actually had a pretty high degree of success on my data set (game tiles).

Re: Detecting duplicate images with Python

#6

Post by danbruc » Thu, Apr 03, 2014, 12:03 PM UTC

ivoflipse wrote: post #4

rplnt wrote: post #2

I feel like this is doing something that was already done better (probably), but I enjoyed the article anyways. Nicely summarized what, why, and how.

Any idea of what might be a better approach?

Many of the steps are similar to ideas used in the scale-invariant feature transform [1]. See also the original paper [2] and an early use for automatic image stitching [3].

[1] http://en.wikipedia.org/wiki/Scale-invariant_feature_transfo...

[2] http://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf

[3] http://www.cs.bath.ac.uk/brown/papers/ijcv2007.pdf

Re: Detecting duplicate images with Python

#7

Post by nicolasp » Thu, Apr 03, 2014, 12:16 PM UTC

ivoflipse wrote: post #4

rplnt wrote: post #2

I feel like this is doing something that was already done better (probably), but I enjoyed the article anyways. Nicely summarized what, why, and how.

Any idea of what might be a better approach?

A comment in the article mentions libpuzzle[1], which looks interesting. A Python binding also exists: PyPuzzle[2].

[1] http://www.pureftpd.org/project/libpuzzle

[2] https://github.com/ArchangelSDY/PyPuzzle

Re: Detecting duplicate images with Python

#8

Post by clintonc » Thu, Apr 03, 2014, 12:22 PM UTC

ivoflipse wrote: post #4

rplnt wrote: post #2

I feel like this is doing something that was already done better (probably), but I enjoyed the article anyways. Nicely summarized what, why, and how.

Any idea of what might be a better approach?

Googling "perceptual hash" yields http://www.phash.org/

Re: Detecting duplicate images with Python

#9

Post by razius » Thu, Apr 03, 2014, 12:28 PM UTC

ivoflipse wrote: post #4

rplnt wrote: post #2

I feel like this is doing something that was already done better (probably), but I enjoyed the article anyways. Nicely summarized what, why, and how.

Any idea of what might be a better approach?

There is no universal solution, there are a few algorithms like SURF [1], SIFT [2], Winsurf [3], dhash (the one presented here), ahash [4], phash [5] each concentrating on a particular feature of the image and has it's strong points and weak points.

[1] http://en.wikipedia.org/wiki/Scale-invariant_feature_transfo...

[2] http://en.wikipedia.org/wiki/SURF

[3] http://www-db.deis.unibo.it/research/papers/IWOSS99.ABP.pdf

[4] http://www.hackerfactor.com/blog/index.php?/archives/529-Kin...

[5] http://www.hackerfactor.com/blog/index.php?/archives/529-Kin...

Re: Detecting duplicate images with Python

#10

Post by herrherr » Thu, Apr 03, 2014, 12:37 PM UTC

We are currently using perceptual hashes (e.g. phash.org) to do hundreds of thousands of image comparisons per day.

As mentioned in another comment, you really have to test different hashing algorithms to find one that suits your needs best. In general though, I think it is in most cases not necessary to develop an algorithm from scratch :)

For us, the much more challenging part was/is to develop a system that can find similar images quickly. If you are interested in things like that have a look at VP/MVP data structures (e.g. http://pnylab.com/pny/papers/vptree/vptree/, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.43.7...).