Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Are Lucene/Solr/ES Still Used for Search?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 223 posts

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#51

Post by aeyes »

crawdog wrote: post #27

Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…

* Fast - bought by Microsoft (Also known as the Enron of Norway...)

That one was painful to live through, we got forced to migrate to Windows and everything went sideways. That was almost 10 years ago with quite a big cluster (tens of nodes).

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#52

Post by bryanrasmussen »

eivarv wrote: post #30
tombert wrote: post #17

Earlier quoted context omitted.

About seven years ago, I got a contracting gig for a website that wanted a "search engine". I remember thinking "Solr/Lucene is old, not pure-functional, and therefore awful!" and decided to build my own. Somehow I even managed to convince the client that this was a good idea. I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was s…

Agreed re: using ES as primary storage (which it is NOT meant as) - as far as I can tell, it might even make you in breach of the GDPR [0]. TLDR: Lucene Delivering a conference talk [1] later this year about it. [0]: https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-... [1]: https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-531...

I think the instructor's response in the linked article is a reasonable defense, you don't really know that data is deleted all the way down to the file level. It is just marked as deleted and could be retrieved by someone clever enough to do so. At some point in the future it will be really deleted.

I don't think the GDPR regulatory agencies are operating at a technical level that they would make an argument that it was not a good enough deletion.

Finally I have to ask this part: assuming ES is not your primary database, how does this get around the GDPR issues? If someone wants their data erased you are supposed to erase it from wherever you store data, I suppose this means ES when it indexes a primary store and finds it has deleted data actually deletes it but if it is told to delete something it keeps it around?

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#54

Post by balfirevic »

eivarv wrote: post #30
tombert wrote: post #17

Earlier quoted context omitted.

About seven years ago, I got a contracting gig for a website that wanted a "search engine". I remember thinking "Solr/Lucene is old, not pure-functional, and therefore awful!" and decided to build my own. Somehow I even managed to convince the client that this was a good idea. I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was s…

Agreed re: using ES as primary storage (which it is NOT meant as) - as far as I can tell, it might even make you in breach of the GDPR [0]. TLDR: Lucene Delivering a conference talk [1] later this year about it. [0]: https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-... [1]: https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-531...

> Agreed re: using ES as primary storage (which it is NOT meant as) - as far as I can tell, it might even make you in breach of the GDPR [0].

How is GDPR compliance of having data in Elastic Search influenced by it being primary vs. secondary storage?

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#55

Post by seesaw »

busterarm wrote: post #33
lm28469 wrote: post #6

Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.

There's folks working on Bleve (written in Go) and developers that I work with want to use it (we use Elasticsearch heavily), but as I've told everyone like you just did, Lucene has a 20 year head start. Thing is, there's heavy demand for something more performant than Elasticsearch, so eventually the market will provide. Meanwhile, Redis Enterprise is trying to grab some 'share with RediSearch, which has some severe…

We use Solr/Lucene heavily, ingesting about 3TB a day. We had to build our own clustering since we started before the Solr cloud project. We have been very happy with the results.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#56

Post by tallanvor »

aeyes wrote: post #51
crawdog wrote: post #27

Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…

* Fast - bought by Microsoft (Also known as the Enron of Norway...) That one was painful to live through, we got forced to migrate to Windows and everything went sideways. That was almost 10 years ago with quite a big cluster (tens of nodes).

You were never forced to migrate to Windows. In fact, the last major customers on ESP were using Linux to the end.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#57

Post by alasano »

james-skemp wrote: post #12

The Sitecore CMS moved from License to Solr for search for on-prem instances. After trying to run it on Windows we were happy there was a third party provider that was easy to work with.

For Sitecore I will shamelessly plug Coveo for Sitecore: https://www.coveo.com/en/solutions/coveo-for-sitecore

I think that we definitely have the best integrated and most full featured solution for Sitecore customers.

It's not only about querying but also having a UI framework, built-in customizable indexing, analytics tracking and access to machine learning available in one package.

Source: Am product manager for Coveo for Sitecore.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#58

Post by deathtrader666 »

lm28469 wrote: post #6

Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.

Wonder what DuckDuckGo might have built for its search..

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#59

Post by devonkim »

showerst wrote: post #50
busterarm wrote: post #46

Earlier quoted context omitted.

> Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. There's a 7-figure/yr Elastic Cloud customer I work w…

I think it depends massively on how much data you have. A great many companies and websites only need a few hundred megs of text data indexed, which is easier to outsource. Once you grow larger than that though, the hosted service prices get astronomical compared to standing up a cluster, assuming you have someone who can admin it.

Things are not quite cut and dry operationally. Some places may have only a few hundred MB of data but they need really high availability and performance (probably picked the wrong solution IMO but still...) which is better guaranteed hosting it yourself and perhaps even outside the cloud. Most places the availability of whatever a hosted solution provides is a better value than spending engineering hours to maintain these things. Devops / SRE folks are expensive compared to most other engineers that deliver features primarily.

My point is that a ton of companies (more for cultural reasons than business requirements IME) are so freaked out by any downtime in any way that they’re going to pay for an engineer to just maintain these things themselves.

Post reply on HN