41–50 of 60 posts
Earlier quoted context omitted.
This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU? What is the scaling like? E.g. what if it was 10 million?
I use Cython (CPU) to brute force 400,000 pHash's of images. It takes somewhere between 100 and 200ms to search.
Earlier quoted context omitted.
Retrieval using CNNs requires computing a cosine similarity matrix. So, for 'n' images, a matrix of size n x n would need to be stored in the memory. As you can see, the storage requirements blow up quadratically as 'n' increases. We have already made some optimizations to reduce the memory footprint but there would be clear upper limits to it (We haven't experimented). As for the numbers, the cifar10 dataset example…
Use approximate nearest neighbor. The FAISS library is good.
Earlier quoted context omitted.
I use Cython (CPU) to brute force 400,000 pHash's of images. It takes somewhere between 100 and 200ms to search.
That sounds good! Would be great if you can share the code, or even better, make a PR to the repo.
cdef extern int __builtin_popcountll(unsigned long long) nogil
dist = __builtin_popcountll(key ^ phash)
It would only take a couple of minutes to fill out the rest.
We've just open-sourced our library imagededup, a Python package that simplifies the task of finding exact and near duplicates in an image collection. It includes several hashing algorithms (PHash, DHash etc) and convolutional neural networks. Secondly, an evaluation framework to judge the quality of deduplication. Finally easy plotting functionality of duplicates and a simple API. We're really excited about this lib…
Did you consider using the PhotoDNA hash algo for finding duplicates? If you’ve heard of it and ruled it out, love to know why. While designed for a very different (and dark) purpose, seems like it might do well for the task.
Could this be used to order a large set of images by similarity?
Could this be used to order a large set of images by similarity?
This came a bit late. I recently decided I had to sort all my photos which I usually just dump in a big photos folder. Using the camera on my phone a lot and also getting a lot of media through whatsapp the collection was getting a bit big. I made a script to calculate the hash of every file and if it found a double it would move it to another duplicate folder. This worked reasonably well but I couldn't stop thinking…
did you evaluate the 'imagehash' [1] library prior to working on this-- any limitations/concerns? the additional CNN seems to be the difference between the two libraries [1] https://github.com/JohannesBuchner/imagehash