Show HN: Imagededup – Finding duplicate images made easy • Hacker News Board

Re: Show HN: Imagededup – Finding duplicate images made easy

#11

Post by throwaway_bad » Mon, Oct 07, 2019, 6:24 AM UTC

sekasi wrote: post #7

Here's a broad and perhaps a bit naive question on this; Reddit, Imgur, and any other site that uploads significant amounts of images from significant amount of users.. do they attempt to do this? To de-dupe images and instead create virtual links? At face value it'd seem like a crazy amount of physical disk space savings, but maybe the processing overhead is too expensive?

People do deduplicate files to save on space, except it's usually based on exact byte match using md5 or sha256. Some don't due to privacy issues: https://news.ycombinator.com/item?id=2438181 (e.g., MPAA can upload all their torrented movies and see which ones uploaded instantly to prove that your system has their copyrighted files)

There's no way to make the UX work out for images that are only similar. Would be pretty wild to upload a picture of myself just to see a picture of my twin used instead.

But I do wonder if it's possible to deduplicate different resolutions of an image that only differ in upscaling/downscaling algorithm and compression level used (thereby solving the jpeg erosion problem: https://xkcd.com/1683/)

Re: Show HN: Imagededup – Finding duplicate images made easy

#12

Post by rcarmo » Mon, Oct 07, 2019, 6:27 AM UTC

This is really nice. I have a semi-abandoned project that uses PHash to canvas my photo library to flag duplicates, and am quite likely to use this instead.

Now if only Apple hadn’t repeatedly broken PyObjC over the years...

Re: Show HN: Imagededup – Finding duplicate images made easy

#13

Post by amrrs » Mon, Oct 07, 2019, 6:36 AM UTC

datitran wrote: post #3

We've just open-sourced our library imagededup, a Python package that simplifies the task of finding exact and near duplicates in an image collection. It includes several hashing algorithms (PHash, DHash etc) and convolutional neural networks. Secondly, an evaluation framework to judge the quality of deduplication. Finally easy plotting functionality of duplicates and a simple API. We're really excited about this lib…

Thanks Dat for open sourcing it, once again I'm shamelessly linking my Kaggle Kernel with `imagededup` installed https://www.kaggle.com/nulldata/image-duplicates-without-dee.... This one used `PHash()` and a run time less than 88 seconds.

Anyone can just simply fork this kernel and add their dataset to it and perform the task!

Re: Show HN: Imagededup – Finding duplicate images made easy

#14

Post by wertenshap » Mon, Oct 07, 2019, 6:55 AM UTC

rcarmo wrote: post #12

This is really nice. I have a semi-abandoned project that uses PHash to canvas my photo library to flag duplicates, and am quite likely to use this instead. Now if only Apple hadn’t repeatedly broken PyObjC over the years...

I’ve been trying to sort through multiple backups of my photo library for several years and such a tool will be very useful.

Re: Show HN: Imagededup – Finding duplicate images made easy

#15

Post by mosselman » Mon, Oct 07, 2019, 7:06 AM UTC

sekasi wrote: post #7

Here's a broad and perhaps a bit naive question on this; Reddit, Imgur, and any other site that uploads significant amounts of images from significant amount of users.. do they attempt to do this? To de-dupe images and instead create virtual links? At face value it'd seem like a crazy amount of physical disk space savings, but maybe the processing overhead is too expensive?

I once built an image comparison feature into some webpage that had uploads. What I did was scale down all images (for comparison only) to something like 100x100 and I think I made them black and white, but I am not sure about that last detail. I'd then XOR one thumbnail with another to compare their level of similarity. I didn't come up with this myself, I put together a few pieces of information from around the web... as with about 100% of things I build ;).

Not perfect, but it worked pretty well for images that were exactly the same. Of course it isn't as advanced as Imagededup.

Re: Show HN: Imagededup – Finding duplicate images made easy

#16

Post by JonathanFly » Mon, Oct 07, 2019, 7:14 AM UTC

Could this be used to order a large set of images by similarity?

Re: Show HN: Imagededup – Finding duplicate images made easy

#17

Post by ggm » Mon, Oct 07, 2019, 7:15 AM UTC

wertenshap wrote: post #14

rcarmo wrote: post #12

This is really nice. I have a semi-abandoned project that uses PHash to canvas my photo library to flag duplicates, and am quite likely to use this instead. Now if only Apple hadn’t repeatedly broken PyObjC over the years...

I’ve been trying to sort through multiple backups of my photo library for several years and such a tool will be very useful.

This too is my usecase. Some backported home from Google takeout, some jpeg recompress, some renamed and date time shifted by jhead. I'd love to be able to group and select the highest Q or pixel count and then canonically order by date.

G

Re: Show HN: Imagededup – Finding duplicate images made easy

#18

Post by cheschire » Mon, Oct 07, 2019, 7:18 AM UTC

sekasi wrote: post #7

Here's a broad and perhaps a bit naive question on this; Reddit, Imgur, and any other site that uploads significant amounts of images from significant amount of users.. do they attempt to do this? To de-dupe images and instead create virtual links? At face value it'd seem like a crazy amount of physical disk space savings, but maybe the processing overhead is too expensive?

The CPU cost would far outweigh the storage cost.

Re: Show HN: Imagededup – Finding duplicate images made easy

#19

Post by tdhoot » Mon, Oct 07, 2019, 7:20 AM UTC

cheschire wrote: post #18

sekasi wrote: post #7

Here's a broad and perhaps a bit naive question on this; Reddit, Imgur, and any other site that uploads significant amounts of images from significant amount of users.. do they attempt to do this? To de-dupe images and instead create virtual links? At face value it'd seem like a crazy amount of physical disk space savings, but maybe the processing overhead is too expensive?

The CPU cost would far outweigh the storage cost.

Only if you go by similarity. If you go by exact dedupes using hashes (which they almost certainly do), the CPU cost is trivial.

Re: Show HN: Imagededup – Finding duplicate images made easy

#20

Post by jeeceebees » Mon, Oct 07, 2019, 7:24 AM UTC

darkmighty wrote: post #4

datitran wrote: post #3

We've just open-sourced our library imagededup, a Python package that simplifies the task of finding exact and near duplicates in an image collection. It includes several hashing algorithms (PHash, DHash etc) and convolutional neural networks. Secondly, an evaluation framework to judge the quality of deduplication. Finally easy plotting functionality of duplicates and a simple API. We're really excited about this lib…

Very nice. If I may ask for clarification on the CNN method (which seems to work quite well), you're taking a CNN that's built for classification but removing the last layer (which I believe is fully connected?) so that you only take the "internal features", is that correct? I would expect some kind of autoencoder would fit here, but very interesting that this works.

This is the same idea that underlies style transfers and metrics like the FID (which is used to judge generative networks' outputs on their similarity to the test set).

The idea is that the activations within an image recognition network are similar for images that are similar and so you can measure the distance between two images in a space that has some semantic meaning.