Hello, author of the post here. Thanks for all your valuable comments. I noticed that there was some talk about neural network driven models. In my follow up post to this one:

http://sujitpal.blogspot.com/2016/06/comparison-of-image-sea...

I describe how I use transfer learning to generate image vectors for my butterfly images from the Caffe reference model trained on ImageNet. There are some other approaches too, but probably not as interesting to this audience.

@GrantS - thanks for the intro and the links, and for pointing out that I was using BoVW incorrectly. Not an image person, trying to get into it from a search/NLP background.

@infinitone and @deckar01 - I measured the goodness of search by checking how many times the query butterfly was the #1 result. I agree its pretty hard to distinguish butterflies from one another, but I needed something to work with, and that was as good as any. I did not want one that crosses subject areas, such as flowers and cars, so a search for a red flower might bring back a red car.

@chrischen - if you got rid of the bucketing portion in my pipeline, I think you might achieve what you are after. But if you are only looking for exact match on color, then hashing might be cheaper.

@rcarmo - thanks for the reference to pHash, I think it might work better in near-duplicate detection kind of cases. Will check it out.