What are some of the ways you can improve on the n-squared nature of these distance calculations at scale? In your use case you have a limited set of hashes you are comparing against, but I'm thinking of the more difficult use case of comparing every image against every other image (the context is ML training and weeding out overly similar examples so that overfitting doesn't happen, not related to CSAM). This quickl…
Hi there! Our example on deduplication [1] takes the case of deduplicating of the Caltech256 [2] dataset for the purpose I think you're describing, which I interpreted as avoiding duplicates in a dataset so that you don't have images that end up in both your training and test sets. Even though the dataset contains >30K images, you can still do this in memory (and find a handful of duplicates!) because each category i…
Re: Show HN: Perceptual hashing tools for detecting child sexual abuse material
#51One more thing: I tried the different hashes that would run on my system (some had a dependency on opencv-contrib that was not met, and which python -m pip couldn't find) and found they aren't what I'm looking for. They may be good at finding different versions of the same image, but what I want is more like environment similarity. So that two pictures taken on the same street would be closer than a picture of the street versus a picture of a forest. They don't seem to be tuned for this task at all. So, my search continues. Let me know if you have suggestions on this as well.