Earlier quoted context omitted.
Deduplicating by SHA (so, exactly the same bytes) is straightforward, and phototprism does that: https://docs.photoprism.app/user-guide/library/duplicates/ Unfortunately, if you've used Google Takeout or other systems that can both downsample your photos and videos, as well as actually deleting _or changing_ metadata, deduplicating becomes a big wad of heuristics. My first approach was to build a UID based on a serie…
I went about depdup in a similar but opposite way then you did - https://github.com/mgolub2/photodb I used libraw to read the actual raw data from my images, ignoring possible metadata that can get changed/updated by Capture One for example. The raw data is just fed into a hashing function, to get an exact content hash. Does not work if your image is down-sampled of course, but that was actually my goal - I want to k…
So if you, say, change the rotation of your RAW image file (which normally will just add or change the EXIF `Orientation` tag), the image content hash will stay the same.
See https://exiftool.org/ExifTool.html#ImageHashType
(of course, your current approach should prove substantively faster, but will only work with the formats that libraw supports)