Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why should I use Elasticsearch instead of building from scratch

news.ycombinator.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#31

Post by sova »

rjkennedy98 wrote: post #11
mlevental wrote: post #6

what a weird question: who looks at a search engine and thinks yea hm that's trivial enough i could do it myself in a weekend?

Seriously, creating an efficient scalable search engine is among the most difficult computer science problems. From stemming, to combined queries, to word tokenization, to handling various string collations, and language issues, and caching, and parallelization of work, and handling huge numbers of writes, there are so many tricky parts. I used to work for a search startup and I can answer the OP's question: do not t…

It depends on the complexity, which language(s) you are using, and how you will parse the search string (which Context Free Grammar it will obey). Just recently, this was a task for me that ended up landing me an interview. A take-home problem. Even a simple search engine needs some clever reverse-indexing for speed. Add any sort of logic like And or OR that not even Google implements and now your parser has to work, and you have to be able to translate from tokenized parse tree with operators to results. It's a great learning exercise for someone with experience, initiative, and enough background with CFG parsing, building a reverse index, and set logic -- but without some key Computer Science building blocks it would end up being quite a challenge.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#33

Post by mlthoughts2018 »

sova wrote: post #30
mlthoughts2018 wrote: post #20

Earlier quoted context omitted.

I worked in a company previously where Solr was used to scale the business, and was not performant for us after a while. We wrote our own search engine at that point. You are right that there are a lot of little “devil in the details” issues. But overall it was a fun experience. This was needed to support some specific machine learning workflows in the search ranking process — which could not be used if we paid the h…

Could you tell us more about the kinds of indices you used, and what you mean by boolean filters? Thank you!

Really it’s not fancy or anything. We used Eigen to represent our normalized bag of words matrix (term-document matrix) as a sparse matrix in CSC and CSR format (which means the data resides in three underlying arrays for the nonzero entries, with indexing conventions for how to use them).

Boolean & multi-choice indices are just companion arrays where position i corresponds to a property of document i in the index: boolean for binary attributes (for example, whether the item has free shipping or not), or using a bigger integer space to encode more options, like say an int8 coupled with helper functions that check which bit is set, maybe for some set of 8 categories the items can be filtered by).

The “index” is just the serialized arrays backing the sparse matrix, the arrays backing the filters, and helper functions for decoding what the filter bits mean.

A query is then just applying the filters followed by performing the sparse matrix inner product and sorting.

It’s very basic, but allows you to heavily optimize it, whether optimizing for deletes, writes, certain heavily used filters, etc.

And you can of course add whatever fancy NLP stuff on top of or in place of the sparse matrix as well.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#34

Post by tedmiston »

mlthoughts2018 wrote: post #20
tedmiston wrote: post #4

You should write one from scratch to get a deeper understanding of how hard it is to return highly relevant results quickly. Tokenizing, stemming, bag of words, and tf-idf for ranking get you to an MVP, but then you realize how good production grade search engines are today. Solr is good. I've been wanting to try Lunr [1] for small sites. [1]: https://github.com/olivernn/lunr.js

I worked in a company previously where Solr was used to scale the business, and was not performant for us after a while. We wrote our own search engine at that point. You are right that there are a lot of little “devil in the details” issues. But overall it was a fun experience. This was needed to support some specific machine learning workflows in the search ranking process — which could not be used if we paid the h…

I realize my comment was a little vague around Solr — my experience is specifically more with Lucene than Solr itself, and I just think of Solr as an easier entry point to Lucene. Did you consider or end building your custom search engine on top of Lucene or something else? What about using a service like Algolia if something like that were available at the time?

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#35

Post by sova »

mlthoughts2018 wrote: post #33
sova wrote: post #30

Earlier quoted context omitted.

Could you tell us more about the kinds of indices you used, and what you mean by boolean filters? Thank you!

Really it’s not fancy or anything. We used Eigen to represent our normalized bag of words matrix (term-document matrix) as a sparse matrix in CSC and CSR format (which means the data resides in three underlying arrays for the nonzero entries, with indexing conventions for how to use them). Boolean & multi-choice indices are just companion arrays where position i corresponds to a property of document i in the index: b…

It sounds like a great approach to enable both speed in searching and customizability in indexing -- no way you get to bit level box ticking with conventional means. Thanks for explaining. Bag of Words, last I recall reading, is a statistical method for predicting the next word, so I'm curious how that played out for you.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#36

Post by dozzie »

33degrees wrote: post #8

Elasticsearch is incredibly deep, and highly performant. If all you need is simple full text search then rolling your own can be an interesting exercise, but I can't imagine the amount of hours it would take to replicate the features I use on a daily basis.

> Elasticsearch is incredibly deep, and highly performant

For some value of "highly performant". I remember its search (exact substring match) being significantly slower than simply running grep on the same data (JSON documents produced from syslog logs) stored in flat files.

It did have several advantages over grep in that scenario (e.g. having a structured language and being accessible for other programs through network), but performance was not one of them.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#37

Post by mlthoughts2018 »

sova wrote: post #35
mlthoughts2018 wrote: post #33

Earlier quoted context omitted.

Really it’s not fancy or anything. We used Eigen to represent our normalized bag of words matrix (term-document matrix) as a sparse matrix in CSC and CSR format (which means the data resides in three underlying arrays for the nonzero entries, with indexing conventions for how to use them). Boolean & multi-choice indices are just companion arrays where position i corresponds to a property of document i in the index: b…

It sounds like a great approach to enable both speed in searching and customizability in indexing -- no way you get to bit level box ticking with conventional means. Thanks for explaining. Bag of Words, last I recall reading, is a statistical method for predicting the next word, so I'm curious how that played out for you.

Bag of words is an encoding method for converting some sequence of words into a long, sparse vector of word counts. The vector has one component for each possible word of the vocabulary, and if the count of word i is N, then you store N in conponent i of the bag of words vector.

If you think of these as sparse row vectors (the columns correspond to all vocabulary entries), then you store them as a matrix where you stack on another row for each “document” in your data set.

Later on when you get a new “document” at query time, you transform it into the same bag of words vector format, and then an inner product between the matrix and the query vector corresponds to a type of relevance / similarity useful for sorting into a ranked order of results.

In practical situations you have to work harder, because you need more units of text than just words (such as n-grams), and the raw term counts usually need to be weighted (e.g. matching a 3-gram probably means more than matching a single word) or normalized (e.g. longer documents happen to have more words, but that doesn’t mean they are more similar), and you need to account for results that are historically more popular or results that are newer.

It’s a very simple approach to document search, but it works well and there are extensions that utilize word embeddings or models that predict rankings of results.

Once you get a system running with the term-document matrix, it is a nice platform for more advanced experimentation and machine learning feature development.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#38

Post by 33degrees »

dozzie wrote: post #36
33degrees wrote: post #8

Elasticsearch is incredibly deep, and highly performant. If all you need is simple full text search then rolling your own can be an interesting exercise, but I can't imagine the amount of hours it would take to replicate the features I use on a daily basis.

> Elasticsearch is incredibly deep, and highly performant For some value of "highly performant". I remember its search (exact substring match) being significantly slower than simply running grep on the same data (JSON documents produced from syslog logs) stored in flat files. It did have several advantages over grep in that scenario (e.g. having a structured language and being accessible for other programs through ne…

Right: my experience is with a lot more complex scenarios, and in comparison to rdbms. Things that would take multiple queries, like aggregations, can be done in a single, fast, call. It does require a proper indexing setup and some fine tuning though.
Post reply on HN