Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Imagededup – Finding duplicate images made easy

github.com

Post reply on HN
Print view View original

21–30 of 60 posts

Re: Show HN: Imagededup – Finding duplicate images made easy

#21

Post by albertzeyer »

This looks nice, and seems to support the most common methods for fingerprinting/hashing. This comes with some heavy dependencies, though (which is reasonable):

    install_requires=[
        'numpy==1.16.3',
        'Pillow==6.0.0',
        'PyWavelets==1.0.3',
        'scipy==1.2.1',
        'tensorflow==2.0.0',
        'tqdm==4.35.0',
        'scikit-learn==0.21.2',
        'matplotlib==3.1.1',
    ],
A while ago, I asked about sth like this (or more about the underlying methods) here on SO:

https://stackoverflow.com/questions/4196453/simple-and-fast-...

There are some interesting discussions. (Nowadays, such a question would have been closed...)

Re: Show HN: Imagededup – Finding duplicate images made easy

#23

Post by tanujjain »

mkl wrote: post #10
datitran wrote: post #3

We've just open-sourced our library imagededup, a Python package that simplifies the task of finding exact and near duplicates in an image collection. It includes several hashing algorithms (PHash, DHash etc) and convolutional neural networks. Secondly, an evaluation framework to judge the quality of deduplication. Finally easy plotting functionality of duplicates and a simple API. We're really excited about this lib…

This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU? What is the scaling like? E.g. what if it was 10 million?

Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements, following are the current considerations: 1. Hashing methods: Generation of hashes is quick (a couple of seconds on about 10K images). The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes. (I would refrain from giving exact numbers since this was done on a local system, not a very good environment for benchmarking) 2. CNN: Here, the time consuming part is encoding generation, which, in the absence of GPU would take much more time (a couple of minutes on 10k images). The retrieval part is pretty quick, but requires memory. So, at this point, using this package on a scale of more than a couple of thousand images is not a good idea when done locally. We would however, address the scale aspect of the problem in future releases. Thanks for your question.

Re: Show HN: Imagededup – Finding duplicate images made easy

#24

Post by flud »

tanujjain wrote: post #23
mkl wrote: post #10

Earlier quoted context omitted.

This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU? What is the scaling like? E.g. what if it was 10 million?

Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements, following are the current considerations: 1. Hashing methods: Generation of hashes is quick (a couple of seconds on about 10K images). The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes. (I would r…

Doesn't sound too bad. But can you elaborate on the last part, about retrieval requiring memory and not scaling to more than a couple of thousands images?

How much memory would you need for ~2000 images, how slow does it get, etc.

Thx

Re: Show HN: Imagededup – Finding duplicate images made easy

#25

Post by jonatron »

mkl wrote: post #10
datitran wrote: post #3

We've just open-sourced our library imagededup, a Python package that simplifies the task of finding exact and near duplicates in an image collection. It includes several hashing algorithms (PHash, DHash etc) and convolutional neural networks. Secondly, an evaluation framework to judge the quality of deduplication. Finally easy plotting functionality of duplicates and a simple API. We're really excited about this lib…

This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU? What is the scaling like? E.g. what if it was 10 million?

I use Cython (CPU) to brute force 400,000 pHash's of images. It takes somewhere between 100 and 200ms to search.

Re: Show HN: Imagededup – Finding duplicate images made easy

#26

Post by orf »

albertzeyer wrote: post #21

This looks nice, and seems to support the most common methods for fingerprinting/hashing. This comes with some heavy dependencies, though (which is reasonable): install_requires=[ 'numpy==1.16.3', 'Pillow==6.0.0', 'PyWavelets==1.0.3', 'scipy==1.2.1', 'tensorflow==2.0.0', 'tqdm==4.35.0', 'scikit-learn==0.21.2', 'matplotlib==3.1.1', ], A while ago, I asked about sth like this (or more about the underlying methods) here…

If this is a library then locking those dependencies down is not great. Does it really need Pillow 6.0.0, and not Pillow 6.0.1?

As this is being consumed by larger applications that may have dependencies that conflict with these, they should be much more liberal.

https://github.com/idealo/imagededup/pull/36

Re: Show HN: Imagededup – Finding duplicate images made easy

#27

Post by mkl »

jonatron wrote: post #25
mkl wrote: post #10

Earlier quoted context omitted.

This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU? What is the scaling like? E.g. what if it was 10 million?

I use Cython (CPU) to brute force 400,000 pHash's of images. It takes somewhere between 100 and 200ms to search.

How long did the generating take?

Re: Show HN: Imagededup – Finding duplicate images made easy

#28

Post by tanujjain »

flud wrote: post #24
tanujjain wrote: post #23

Earlier quoted context omitted.

Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements, following are the current considerations: 1. Hashing methods: Generation of hashes is quick (a couple of seconds on about 10K images). The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes. (I would r…

Doesn't sound too bad. But can you elaborate on the last part, about retrieval requiring memory and not scaling to more than a couple of thousands images? How much memory would you need for ~2000 images, how slow does it get, etc. Thx

Retrieval using CNNs requires computing a cosine similarity matrix. So, for 'n' images, a matrix of size n x n would need to be stored in the memory. As you can see, the storage requirements blow up quadratically as 'n' increases. We have already made some optimizations to reduce the memory footprint but there would be clear upper limits to it (We haven't experimented). As for the numbers, the cifar10 dataset example present in the repo was carried out on google colab notebook. The dataset has 60k images and ended up consuming about 6G of RAM using CNN. However, since there hasn't yet been a principled benchmarking done on the package, I would consider these numbers as only marginally indicative of the capabilities. A better way to figure out if it works for you would be to try it out with your own dataset, starting out at a small scale and increasing the dataset size gradually.

Re: Show HN: Imagededup – Finding duplicate images made easy

#29

Post by shifto »

This came a bit late. I recently decided I had to sort all my photos which I usually just dump in a big photos folder. Using the camera on my phone a lot and also getting a lot of media through whatsapp the collection was getting a bit big.

I made a script to calculate the hash of every file and if it found a double it would move it to another duplicate folder. This worked reasonably well but I couldn't stop thinking there should be more than 1 solution already made for this.

Re: Show HN: Imagededup – Finding duplicate images made easy

#30

Post by jonatron »

mkl wrote: post #27
jonatron wrote: post #25

Earlier quoted context omitted.

I use Cython (CPU) to brute force 400,000 pHash's of images. It takes somewhere between 100 and 200ms to search.

How long did the generating take?

We store pHash'es in a database, but just quickly checking on my laptop, between 1-2ms to generate a single image pHash. IO could be significant for many images.
Post reply on HN