Ask HN: Why should I use Elasticsearch instead of building from scratch • Hacker News Board

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#11

Post by rjkennedy98 » Sun, Jun 10, 2018, 1:33 PM UTC

mlevental wrote: post #6

what a weird question: who looks at a search engine and thinks yea hm that's trivial enough i could do it myself in a weekend?

Seriously, creating an efficient scalable search engine is among the most difficult computer science problems. From stemming, to combined queries, to word tokenization, to handling various string collations, and language issues, and caching, and parallelization of work, and handling huge numbers of writes, there are so many tricky parts. I used to work for a search startup and I can answer the OP's question: do not try to write your own search engine. That work should be done by someone with a PhD and decades of experience. Even elastic which is great software has issues, such as not being transactional, having issues handling huge numbers of writes, ect.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#12

Post by danielecook » Sun, Jun 10, 2018, 1:38 PM UTC

dewey wrote: post #7

With that little information from the OP the answer is probably: Use ES or if it's a small side project use your DB's included full text search if it's good enough.

When is a full text search within a DB not good enough? Is ES usually used along side a typical RDS or is it a replacement?

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#13

Post by based2 » Sun, Jun 10, 2018, 1:39 PM UTC

What do you want ?

https://db-engines.com/en/ranking

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#14

Post by dajohnson89 » Sun, Jun 10, 2018, 1:41 PM UTC

ian1321 wrote: post #9

Tough to answer w/o more info. FWIW, I've used Lucene, Solr, and Elasticsearch and have ended up settling on Lucene being the best interface for me.

I thought Lucene was the underlying query language, whereas Solr & ES just utilized both...

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#15

Post by mistrial9 » Sun, Jun 10, 2018, 1:42 PM UTC

brightball wrote: post #10

dewey wrote: post #7

With that little information from the OP the answer is probably: Use ES or if it's a small side project use your DB's included full text search if it's good enough.

You can go far using purely Postgres full text search. I ported a site from Solr to PG full text because of strange syncing issues and it was just as fast. Afterwards I never really saw the point of any of the search systems other than elastic search because the streaming capabilities that it gives you.

agree, and add your own caching. Lots of pop in the web world comes from layers of well-built caching.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#16

Post by courtneycouch0 » Sun, Jun 10, 2018, 1:43 PM UTC

Definitely build it completely from scratch. You should roll your own TCP libraries as well. Don't trust anything you didn't write yourself. Come to think of it, I'm not sure you should rely on someone else's hardware either.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#17

Post by dewey » Sun, Jun 10, 2018, 1:43 PM UTC

danielecook wrote: post #12

dewey wrote: post #7

With that little information from the OP the answer is probably: Use ES or if it's a small side project use your DB's included full text search if it's good enough.

When is a full text search within a DB not good enough? Is ES usually used along side a typical RDS or is it a replacement?

I don't have a lot of experience with PG full text search in production or a bigger scale. I'd just suspected that it doesn't perform that well if you need a lot of filters, range queries etc? Maybe someone with more experience can chime in.

At work we just materialize the data from PG into ES and take advantage of the powerful ES queries and redundancy. Scaling up by just adding nodes is easier.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#18

Post by rjkennedy98 » Sun, Jun 10, 2018, 1:44 PM UTC

danielecook wrote: post #12

dewey wrote: post #7

With that little information from the OP the answer is probably: Use ES or if it's a small side project use your DB's included full text search if it's good enough.

When is a full text search within a DB not good enough? Is ES usually used along side a typical RDS or is it a replacement?

Depends on the use case. We use it alongside Cassandra and MySQL, but we also deal with tens of billions of records. If you just have a webblog, you could just use elastic.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#19

Post by ctvo » Sun, Jun 10, 2018, 1:46 PM UTC

When did this place become such a low effort brain dump? I can't believe I'm so annoyed by this question. Google. Make some educated trade-off decisions based on your context.

Use HN to poll for opinions and experiences from others, not for things that take 30 minutes to resolve.

Re: Ask HN: Why should I use Elasticsearch instead of building from scratch

#20

Post by mlthoughts2018 » Sun, Jun 10, 2018, 1:46 PM UTC

tedmiston wrote: post #4

You should write one from scratch to get a deeper understanding of how hard it is to return highly relevant results quickly. Tokenizing, stemming, bag of words, and tf-idf for ranking get you to an MVP, but then you realize how good production grade search engines are today. Solr is good. I've been wanting to try Lunr [1] for small sites. [1]: https://github.com/olivernn/lunr.js

I worked in a company previously where Solr was used to scale the business, and was not performant for us after a while.

We wrote our own search engine at that point. You are right that there are a lot of little “devil in the details” issues. But overall it was a fun experience.

This was needed to support some specific machine learning workflows in the search ranking process — which could not be used if we paid the high latency cost to first get preliminary results in Solr.

So we took a “create your own index data structures” approach with index data (both the normalized bag of words vectors and companion data like boolean filters), which allowed us to highly optimize the initial broad ranking query. Latency was low enough that it allowed the time cost of calling follow-on machine learning services.

This was for a fairly high-traffic product search engine at an online retailer. It ended up working very well and over a span of about two years we eventually rolled all search traffic onto the in-house platform, even the parts not needing the machine learning services, and our query latencies went down across all our traffic, and we retired the original Solr implementation.

Wouldn’t be the right choice for everyone, but it informs my opinion a lot about the worthwhileness of creating an in-house search engine to specifically replace Solr. I’d suspect a lot of medium-sized or large companies running Solr should seriously consider it.