Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch Based Image Search Using RGB Signatures

sujitpal.blogspot.com

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Elasticsearch Based Image Search Using RGB Signatures

#11

Post by jkldotio »

infinitone wrote: post #4
GrantS wrote: post #3

For anyone interested in the computer vision side of this topic, the author here is using a variant of color histograms, which was state of the art around 1990 [1][2]. Since 2003, bag of visual words approaches have usually meant extracting SIFT-like features from a database of images, quantizing the features down to a list of thousands or millions of "words", and then treating the images like documents containing th…

Forgive my ignorance, but is there something like word2vec but for images- like an image2vec? In terms of text processing, word2vec is one of the best approaches. So can't you describe the features of images using neural networks and then vectorize them using word2vec? Its been acouple years since my computer vision course (was my favorite course in university) but isn't SIFT a bit '99? Aren't there better methods no…

I think in addition to histograms and features like SIFT, SURF, and others like DAISY[0] that would permit searching with images as a query it would be beneficial just to use a neural network to text index using the classes at very top of a neural network, although you are right some features in layers just below that could be used too as I understand it.

You could then use classical text indexing on the text, perhaps with a topic model like LDA. Then an image with a plane in it will be indexed by "plane" via the output of the neural network but would also come first, or in the top results, when using "flight" as the query via a topic model.

Ditto for word2vec or para2vec over those words, the benefit being you can bring the knowledge of relations contained in the textual training data, Wikipedia or something else, to bear on the problem. I.e. a golf club and a baseball glove might not be correlated in the neural network that annotated the images but might be correlated in the text based knowledge model trained on Wikipedia and so a query of "sport" might bring both images up.

The big players like Google already have caption generation that's capturing relationships between objects.[1]

[0]http://scikit-image.org/docs/dev/auto_examples/features_dete... [1]https://arxiv.org/pdf/1411.4555.pdf

Re: Elasticsearch Based Image Search Using RGB Signatures

#12

Post by UncleChis »

infinitone wrote: post #4
GrantS wrote: post #3

For anyone interested in the computer vision side of this topic, the author here is using a variant of color histograms, which was state of the art around 1990 [1][2]. Since 2003, bag of visual words approaches have usually meant extracting SIFT-like features from a database of images, quantizing the features down to a list of thousands or millions of "words", and then treating the images like documents containing th…

Forgive my ignorance, but is there something like word2vec but for images- like an image2vec? In terms of text processing, word2vec is one of the best approaches. So can't you describe the features of images using neural networks and then vectorize them using word2vec? Its been acouple years since my computer vision course (was my favorite course in university) but isn't SIFT a bit '99? Aren't there better methods no…

There're ton more local features proposed since SIFT but SIFT, arguably, still among the best. Deep feature, on the other hand, is more like global feature. May be a combination of these two could give the best?

Re: Elasticsearch Based Image Search Using RGB Signatures

#13

Post by UncleChis »

I have put quite a bit of similar effort to image retrieval using Elasticsearch before. While it is nice and convenient, what I found was that Elastic Search is too slow for larger scale (million of Images with dictionary of millions visual words in BoW model). May be there're some steps that I did not do right, but I gave up.

Re: Elasticsearch Based Image Search Using RGB Signatures

#15

Post by jerluc »

In my [limited] experience with CBIR and image based search, I found that using a color space with perceptual spatial qualities (such as one of the CIE LaB variants) to be more effective than a purely normalized geometric color space (such as RGB or HSV), as color similarities in the latter may not make much sense to a human.

Re: Elasticsearch Based Image Search Using RGB Signatures

#16

Post by Ronsenshi »

infinitone wrote: post #4
GrantS wrote: post #3

For anyone interested in the computer vision side of this topic, the author here is using a variant of color histograms, which was state of the art around 1990 [1][2]. Since 2003, bag of visual words approaches have usually meant extracting SIFT-like features from a database of images, quantizing the features down to a list of thousands or millions of "words", and then treating the images like documents containing th…

Forgive my ignorance, but is there something like word2vec but for images- like an image2vec? In terms of text processing, word2vec is one of the best approaches. So can't you describe the features of images using neural networks and then vectorize them using word2vec? Its been acouple years since my computer vision course (was my favorite course in university) but isn't SIFT a bit '99? Aren't there better methods no…

I think depending on the need Neural Network feature detection/description might be an overkill.

What is great about stuff like SIFT and more modern ORB, BRISK and AKAZE is that they are fast and given appropriate implementation they would work just as well as Neural Network would. I haven't researched NN computer vision whole lot, but it seems like it might be slower in feature detection/description compared to traditional approach. If that's the case then for live/near-live video processing NN won't work that well.

Re: Elasticsearch Based Image Search Using RGB Signatures

#17

Post by apstls »

infinitone wrote: post #4
GrantS wrote: post #3

For anyone interested in the computer vision side of this topic, the author here is using a variant of color histograms, which was state of the art around 1990 [1][2]. Since 2003, bag of visual words approaches have usually meant extracting SIFT-like features from a database of images, quantizing the features down to a list of thousands or millions of "words", and then treating the images like documents containing th…

Forgive my ignorance, but is there something like word2vec but for images- like an image2vec? In terms of text processing, word2vec is one of the best approaches. So can't you describe the features of images using neural networks and then vectorize them using word2vec? Its been acouple years since my computer vision course (was my favorite course in university) but isn't SIFT a bit '99? Aren't there better methods no…

Recent research in deep manifold traversal may interest you [1]. This method uses deep neural networks to approximate the manifold of natural images, learns transformations that traverse the manifold from one image class to another (i.e. from images of young people to images of old people), and then allows you to transform any source image into a new, automatically-generated image by mapping it to the manifold and performing the transformation. So, for example, the paper's authors learn the transformation from images of young people to old people, and then are able to generate realistic-looking images of the celebrities with older facial features (as well as darken hair colors, change skin tones, and even colorize black and white images). This is somewhat analogous to the way that word2vec vectors allow you to do things like queen = king + (man - woman), with an image's projection onto the approximated manifold being analogous to a word's word2vec vector.

[1] http://arxiv.org/pdf/1511.06421v3.pdf

Re: Elasticsearch Based Image Search Using RGB Signatures

#18

Post by Ronsenshi »

GrantS wrote: post #3

For anyone interested in the computer vision side of this topic, the author here is using a variant of color histograms, which was state of the art around 1990 [1][2]. Since 2003, bag of visual words approaches have usually meant extracting SIFT-like features from a database of images, quantizing the features down to a list of thousands or millions of "words", and then treating the images like documents containing th…

For large datasets simple "Bag of Words" approach actually is not that great since for given set of features you have to compare it to the whole vocabulary. More modern approach calls for use of Vocabulary Tree which represents your bag of words. This vocabulary tree significantly reduces amount of matching that has to be done for each individual feature.

[1] http://www.cs.utexas.edu/~grauman/courses/fall2009/papers/ba...

Post reply on HN