Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Imagededup – Finding duplicate images made easy

github.com

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Show HN: Imagededup – Finding duplicate images made easy

#31

Post by jayalpha »

flud wrote: post #24
tanujjain wrote: post #23

Earlier quoted context omitted.

Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements, following are the current considerations: 1. Hashing methods: Generation of hashes is quick (a couple of seconds on about 10K images). The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes. (I would r…

Doesn't sound too bad. But can you elaborate on the last part, about retrieval requiring memory and not scaling to more than a couple of thousands images? How much memory would you need for ~2000 images, how slow does it get, etc. Thx

"The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes"

"Doesn't sound too bad."

I doesn't? As a word of caution: If a 10k dataset takes a few minutes I would be careful how long 10MM pictures take. I predict it does not take 10MM/10k times a few minutes.

Re: Show HN: Imagededup – Finding duplicate images made easy

#32

Post by jayalpha »

Roritharr wrote: post #6
datitran wrote: post #3

We've just open-sourced our library imagededup, a Python package that simplifies the task of finding exact and near duplicates in an image collection. It includes several hashing algorithms (PHash, DHash etc) and convolutional neural networks. Secondly, an evaluation framework to judge the quality of deduplication. Finally easy plotting functionality of duplicates and a simple API. We're really excited about this lib…

Thanks! This is a very important problemspace for us, as we do document processing and double/triple uploads are hard to detect and create unnecessary effort. Can you elaborate a little bit on the performance you've observed? Can it work iteratively, basically asking one image at a time "have we already seen this?" Would it work for text documents of varying quality aswell or is this unfeasible?

Do you use OCR on the documents?

Re: Show HN: Imagededup – Finding duplicate images made easy

#33

Post by bonoboTP »

heipei wrote: post #8
darkmighty wrote: post #4

Earlier quoted context omitted.

Very nice. If I may ask for clarification on the CNN method (which seems to work quite well), you're taking a CNN that's built for classification but removing the last layer (which I believe is fully connected?) so that you only take the "internal features", is that correct? I would expect some kind of autoencoder would fit here, but very interesting that this works.

This is the concept behind "transfer learning", taking a fully-trained CNN (ResNet, Inception, MobileNet, etc) and removing the last or last two layers. What I don't understand yet is when to use which trained network, i.e. if they have different features that make them suited for different application domains.

I don't think there's a lot to say about this, you want the pretraining on a large dataset and on a task that's similar to yours. Probably largeness is somewhat more important. These things aren't an exact science at this point.

Re: Show HN: Imagededup – Finding duplicate images made easy

#34

Post by dgacmu »

tanujjain wrote: post #28
flud wrote: post #24

Earlier quoted context omitted.

Doesn't sound too bad. But can you elaborate on the last part, about retrieval requiring memory and not scaling to more than a couple of thousands images? How much memory would you need for ~2000 images, how slow does it get, etc. Thx

Retrieval using CNNs requires computing a cosine similarity matrix. So, for 'n' images, a matrix of size n x n would need to be stored in the memory. As you can see, the storage requirements blow up quadratically as 'n' increases. We have already made some optimizations to reduce the memory footprint but there would be clear upper limits to it (We haven't experimented). As for the numbers, the cifar10 dataset example…

Use approximate nearest neighbor. The FAISS library is good.

Re: Show HN: Imagededup – Finding duplicate images made easy

#35

Post by donatj »

For a split second I was excited and terrified because I thought this was a very similarly named Go project I wrote a while ago. It’s nowhere near as fancy but is very fast and has a very similar name.

I don’t have the background in imaging these people likely have but mine works by breaking an image into an X by X map of average colors and comparing, written specifically because I needed to find similar images of different aspect ratios and at the time I couldn’t find anything.

https://github.com/donatj/imgdedup

Re: Show HN: Imagededup – Finding duplicate images made easy

#37

Post by jonatron »

tanujjain wrote: post #28
flud wrote: post #24

Earlier quoted context omitted.

Doesn't sound too bad. But can you elaborate on the last part, about retrieval requiring memory and not scaling to more than a couple of thousands images? How much memory would you need for ~2000 images, how slow does it get, etc. Thx

Retrieval using CNNs requires computing a cosine similarity matrix. So, for 'n' images, a matrix of size n x n would need to be stored in the memory. As you can see, the storage requirements blow up quadratically as 'n' increases. We have already made some optimizations to reduce the memory footprint but there would be clear upper limits to it (We haven't experimented). As for the numbers, the cifar10 dataset example…

Couldn't you add images>threshold to a dict/map as you iterate, rather than building a complete matrix, then iterating through that?

Re: Show HN: Imagededup – Finding duplicate images made easy

#38

Post by jsjohnst »

datitran wrote: post #3

We've just open-sourced our library imagededup, a Python package that simplifies the task of finding exact and near duplicates in an image collection. It includes several hashing algorithms (PHash, DHash etc) and convolutional neural networks. Secondly, an evaluation framework to judge the quality of deduplication. Finally easy plotting functionality of duplicates and a simple API. We're really excited about this lib…

Did you consider using the PhotoDNA hash algo for finding duplicates? If you’ve heard of it and ruled it out, love to know why. While designed for a very different (and dark) purpose, seems like it might do well for the task.

Re: Show HN: Imagededup – Finding duplicate images made easy

#39

Post by jsjohnst »

sekasi wrote: post #7

Here's a broad and perhaps a bit naive question on this; Reddit, Imgur, and any other site that uploads significant amounts of images from significant amount of users.. do they attempt to do this? To de-dupe images and instead create virtual links? At face value it'd seem like a crazy amount of physical disk space savings, but maybe the processing overhead is too expensive?

Tumblr did something similar, but only for exact matches. You can tell if it’s a legacy image or not by looking for a hash in the image url path.

Legacy style: https://66.media.tumblr.com/tumblr_m61cvzNYF81qg0jdoo1_640.g...

New style: https://66.media.tumblr.com/76451d8fee12cd3c5971e20bb8e236e3...

Re: Show HN: Imagededup – Finding duplicate images made easy

#40

Post by tanujjain »

jonatron wrote: post #37
tanujjain wrote: post #28

Earlier quoted context omitted.

Retrieval using CNNs requires computing a cosine similarity matrix. So, for 'n' images, a matrix of size n x n would need to be stored in the memory. As you can see, the storage requirements blow up quadratically as 'n' increases. We have already made some optimizations to reduce the memory footprint but there would be clear upper limits to it (We haven't experimented). As for the numbers, the cifar10 dataset example…

Couldn't you add images>threshold to a dict/map as you iterate, rather than building a complete matrix, then iterating through that?

We tried that approach, but it was way too slow.
Post reply on HN