The simple answer is because Elasticsearch has had thousand of hours already put into writing its code base. The real question should be why shouldn't you use Elasticsearch? Is the code base to large to fit where you need it to be? Will it be able to scale with your project? Is it efficient enough for your requirements? When looking to use a piece of technology the requirements and long term effects are what matter. Roll your own if that is what is required for you to reach your end goal.
Ask HN: Why should I use Elasticsearch instead of building from scratch
21–30 of 39 posts
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#22Earlier quoted context omitted.
When is a full text search within a DB not good enough? Is ES usually used along side a typical RDS or is it a replacement?
I don't have a lot of experience with PG full text search in production or a bigger scale. I'd just suspected that it doesn't perform that well if you need a lot of filters, range queries etc? Maybe someone with more experience can chime in. At work we just materialize the data from PG into ES and take advantage of the powerful ES queries and redundancy. Scaling up by just adding nodes is easier.
How do you verify that all the data moved into ES from pg and what got lost in transit.
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#23Tough to answer w/o more info. FWIW, I've used Lucene, Solr, and Elasticsearch and have ended up settling on Lucene being the best interface for me.
I thought Lucene was the underlying query language, whereas Solr & ES just utilized both...
Yeah I'm not sure what the OP is talking about. Lucene is the java search library that Elastic uses. Elastic is a full clustered search engine with HA, sharding, and a rest api. They aren't exactly interchangeable.
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#24Actually, the origin story of Elasticsearch started with Shay Banon attempting to build a cooking app for his wife who was a chef.
> JAXenter: You started Compass, your first Lucene-based technology, in 2004. Do you remember how and why you became interested in Lucene in the first place?
> Shay Banon: Reminiscing on Compass birth always puts a smile on my face. Compass, and my involvement with Lucene, started by chance. At the time, I was a newlywed that just moved to London to support my wife with her dream of becoming a chef. I was unemployed, and desperately in need of a job, so I decided to play around with “new age” technologies in order to get my skills more uptodate. Playing around with new technologies only works when you are actually trying to build something, so I decided to build an app that my wife could use to capture all the cooking knowledge she was gathering during her chef lessons.
> I picked many different technologies for this cooking app, but at the core of it, in my mind, was a single search box where the cooking knowledge experience would start a single box where typing a concept, a thought, or an ingredient would start the path towards exploring what was possible.
> This quickly led me to Lucene, which was the defacto search library available for Java at the time. I got immersed in it, and Compass was born out of the effort of trying to simplify using Lucene in your typical Java applications (conceptually, it simply started as a “Hibernate” (Java ORM library) for Lucene).
> I got completely hooked with the project, and was working on it more than the cooking app itself, up to a point where it was taking most of my time. I decided to open source it a few months afterwards, and it immediately took off. Compass basically allowed users to easily map their domain model (the code that maps app/business concepts in a typical program) to Lucene, easily index them, and then easily search them.
> That freedom caused people to start to use Compass, and Lucene, in situations that were wonderfully unexpected. Imagine already having the model of a Trade in your financial app, one could easily index that Trade using Compass into Lucene, and then search for it. The freedom of searching across any aspect of a Trade allowed users to convey this freedom to their users, which proved to be an extremely powerful concept.
> Effectively, this allowed me to be in the front seat of talking and working with actual users that were discovering, as was I, the amazing power that search can have when it comes to delivering business value to their users. Oh, and btw, my wife is still waiting for that cooking app. Now, 10 years later, it is the basis of Elasticsearch.
https://jaxenter.com/elasticsearch-founder-interview-112677....
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#25When did this place become such a low effort brain dump? I can't believe I'm so annoyed by this question. Google. Make some educated trade-off decisions based on your context. Use HN to poll for opinions and experiences from others, not for things that take 30 minutes to resolve.
As much OP did lazy work by posting this question, I don’t think your response was fair or helpful in any way. In the end you didn’t add anything to the conversation either.
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#26Earlier quoted context omitted.
I don't have a lot of experience with PG full text search in production or a bigger scale. I'd just suspected that it doesn't perform that well if you need a lot of filters, range queries etc? Maybe someone with more experience can chime in. At work we just materialize the data from PG into ES and take advantage of the powerful ES queries and redundancy. Scaling up by just adding nodes is easier.
How do you verify that all the data moved into ES from pg and what got lost in transit.
For our use case we don't really need a strict verification. Everything that should go to ES is put in a queue (using DB triggers) and then sent off to ES. If something should cause errors we'd see that by observing the error rate on ES ingest I assume.
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#27When did this place become such a low effort brain dump? I can't believe I'm so annoyed by this question. Google. Make some educated trade-off decisions based on your context. Use HN to poll for opinions and experiences from others, not for things that take 30 minutes to resolve.
I tend to agree with you, however google isn’t as straightforward as it used to be. Every developer and their mom has a blog these days, or a brand, whatever. Sometimes just cross referencing fundamentals and bonafides is a time sink.
That said, this question probably works better somewhere like Reddit programming or some IIRC where Elasticsearch hangs out. After going that route, it’s probably fine to poll here based on that research.
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#28When did this place become such a low effort brain dump? I can't believe I'm so annoyed by this question. Google. Make some educated trade-off decisions based on your context. Use HN to poll for opinions and experiences from others, not for things that take 30 minutes to resolve.
As much OP did lazy work by posting this question, I don’t think your response was fair or helpful in any way. In the end you didn’t add anything to the conversation either.
What can you add to this discussion? 0 context is provided. It's not even a properly formed question. As someone who's familiar with Lucene, Solr and ElasticSearch even if I wanted to help I can't.
Sometimes posts are shit, and it's OK to call it out to hopefully improve this site collectively.
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#29Earlier quoted context omitted.
You can go far using purely Postgres full text search. I ported a site from Solr to PG full text because of strange syncing issues and it was just as fast. Afterwards I never really saw the point of any of the search systems other than elastic search because the streaming capabilities that it gives you.
agree, and add your own caching. Lots of pop in the web world comes from layers of well-built caching.
This is probably true in many more ways than I am aware of. Do you have any other good examples or resources to suggest?
Re: Ask HN: Why should I use Elasticsearch instead of building from scratch
#30You should write one from scratch to get a deeper understanding of how hard it is to return highly relevant results quickly. Tokenizing, stemming, bag of words, and tf-idf for ranking get you to an MVP, but then you realize how good production grade search engines are today. Solr is good. I've been wanting to try Lunr [1] for small sites. [1]: https://github.com/olivernn/lunr.js
I worked in a company previously where Solr was used to scale the business, and was not performant for us after a while. We wrote our own search engine at that point. You are right that there are a lot of little “devil in the details” issues. But overall it was a fun experience. This was needed to support some specific machine learning workflows in the search ranking process — which could not be used if we paid the h…
Could you tell us more about the kinds of indices you used, and what you mean by boolean filters? Thank you!