Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Imagededup – Finding duplicate images made easy

github.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Show HN: Imagededup – Finding duplicate images made easy

#51

Post by datitran »

orf wrote: post #26
albertzeyer wrote: post #21

This looks nice, and seems to support the most common methods for fingerprinting/hashing. This comes with some heavy dependencies, though (which is reasonable): install_requires=[ 'numpy==1.16.3', 'Pillow==6.0.0', 'PyWavelets==1.0.3', 'scipy==1.2.1', 'tensorflow==2.0.0', 'tqdm==4.35.0', 'scikit-learn==0.21.2', 'matplotlib==3.1.1', ], A while ago, I asked about sth like this (or more about the underlying methods) here…

If this is a library then locking those dependencies down is not great. Does it really need Pillow 6.0.0, and not Pillow 6.0.1? As this is being consumed by larger applications that may have dependencies that conflict with these, they should be much more liberal. https://github.com/idealo/imagededup/pull/36

Yes you're right thanks for pointing this out!

Re: Show HN: Imagededup – Finding duplicate images made easy

#52

Post by gravypod »

tanujjain wrote: post #23
mkl wrote: post #10

Earlier quoted context omitted.

This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU? What is the scaling like? E.g. what if it was 10 million?

Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements, following are the current considerations: 1. Hashing methods: Generation of hashes is quick (a couple of seconds on about 10K images). The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes. (I would r…

What way is the hash represented? Have you looked at NN libraries like faiss [0] and NGT [1]? Those can quite easily handle a nearest neighbor search of 10 million vectors and, from my understanding, they turn their vectors into some kind of hash that is then searched.

[0] - https://github.com/facebookresearch/faiss

[1] - https://github.com/yahoojapan/NGT

Re: Show HN: Imagededup – Finding duplicate images made easy

#53

Post by herohamp »

cosmic_quanta wrote: post #36

Nice project! I wonder how much of it could be adapted to finding duplicate documents, e.g. homeworks, CVs, etc. Presumably, the hashing would have to be adapted slightly. But how much?

For documents I wouldnt do anything listed here. I would just compare the contents on a line by line or word by word basis

Re: Show HN: Imagededup – Finding duplicate images made easy

#55

Post by tanujjain »

gravypod wrote: post #52
tanujjain wrote: post #23

Earlier quoted context omitted.

Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements, following are the current considerations: 1. Hashing methods: Generation of hashes is quick (a couple of seconds on about 10K images). The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes. (I would r…

What way is the hash represented? Have you looked at NN libraries like faiss [0] and NGT [1]? Those can quite easily handle a nearest neighbor search of 10 million vectors and, from my understanding, they turn their vectors into some kind of hash that is then searched. [0] - https://github.com/facebookresearch/faiss [1] - https://github.com/yahoojapan/NGT

The hashes are 16 character hexadecimals represented as strings. Had a quick look at the faiss package and it looks promising. Would consider it for the next versions.

Re: Show HN: Imagededup – Finding duplicate images made easy

#56

Post by tanujjain »

jonatron wrote: post #25
mkl wrote: post #10

Earlier quoted context omitted.

This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU? What is the scaling like? E.g. what if it was 10 million?

I use Cython (CPU) to brute force 400,000 pHash's of images. It takes somewhere between 100 and 200ms to search.

Only realized now that the 100-200ms time you refer to is for a single search and not for 400,000 searches. The package already achieves this brute-force speed. In fact, the package also implements bktree, which, depending upon the distance threshold passed, could drastically reduce the search time. Moreover, the search through bktree is also parallelized in the package(each image's hash gets searched through the tree independently after the tree is constructed). On one of the example dataset containing 10k images, with a distance threshold of 10 (for 64-bit hashes), the retrieval time per image obtained was < 50 ms.

Re: Show HN: Imagededup – Finding duplicate images made easy

#57

Post by tanujjain »

jlg23 wrote: post #45
JonathanFly wrote: post #16

Could this be used to order a large set of images by similarity?

The API does not seem to support this, but it should be easy to hack this (return not only a list of duplicates but the actual distance to the target image along with it, then sort results by distance).

The api already supports returning the hamming distances/cosine similarities along with the duplicate file list which can be used to sort the files. Please refer the docs for 'find_duplicates' function for more.

Re: Show HN: Imagededup – Finding duplicate images made easy

#58

Post by tanujjain »

throwaway_bad wrote: post #11
sekasi wrote: post #7

Here's a broad and perhaps a bit naive question on this; Reddit, Imgur, and any other site that uploads significant amounts of images from significant amount of users.. do they attempt to do this? To de-dupe images and instead create virtual links? At face value it'd seem like a crazy amount of physical disk space savings, but maybe the processing overhead is too expensive?

People do deduplicate files to save on space, except it's usually based on exact byte match using md5 or sha256. Some don't due to privacy issues: https://news.ycombinator.com/item?id=2438181 (e.g., MPAA can upload all their torrented movies and see which ones uploaded instantly to prove that your system has their copyrighted files) There's no way to make the UX work out for images that are only similar . Would be pr…

The cnn methods in the package are particularly robust against resolution differences. In fact, if it's just a simple up/downscale that differentiates 2 images, then even hashing algorithms could be expected to do a good job.

Re: Show HN: Imagededup – Finding duplicate images made easy

#59

Post by gravypod »

tanujjain wrote: post #55
gravypod wrote: post #52

Earlier quoted context omitted.

What way is the hash represented? Have you looked at NN libraries like faiss [0] and NGT [1]? Those can quite easily handle a nearest neighbor search of 10 million vectors and, from my understanding, they turn their vectors into some kind of hash that is then searched. [0] - https://github.com/facebookresearch/faiss [1] - https://github.com/yahoojapan/NGT

The hashes are 16 character hexadecimals represented as strings. Had a quick look at the faiss package and it looks promising. Would consider it for the next versions.

If you're interested in collaboration I'd be happy to help with a prod-focused version. My work has a need for a shardable daemon for dedup tasks. My personal email is in my description and I'm also available via josh@xix.ai.

We also have an image heavy production use case that would be able to yield some nice metrics from this tool.

Re: Show HN: Imagededup – Finding duplicate images made easy

#60

Post by jonatron »

tanujjain wrote: post #56
jonatron wrote: post #25

Earlier quoted context omitted.

I use Cython (CPU) to brute force 400,000 pHash's of images. It takes somewhere between 100 and 200ms to search.

Only realized now that the 100-200ms time you refer to is for a single search and not for 400,000 searches. The package already achieves this brute-force speed. In fact, the package also implements bktree, which, depending upon the distance threshold passed, could drastically reduce the search time. Moreover, the search through bktree is also parallelized in the package(each image's hash gets searched through the tre…

The 100-200ms time I referred to was indeed a single search. The difference is, it's on a single core. Cython definitely makes the hamming distance function faster.
Post reply on HN