Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Are Lucene/Solr/ES Still Used for Search?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 223 posts

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#141

Post by hackbinary »

crawdog wrote: post #27

Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…

We have on prem Coveo, which is based on ES. It is/they are horrible. We pay a large amount in Enterprise support and maintenance, and it is next to nonexistent.

Thankfully we are moving to Azure Cloud Search services, which just work, over the next year.

Good bye manure pile, hello compost.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#145

Post by josh2600 »

aeyes wrote: post #51
crawdog wrote: post #27

Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…

* Fast - bought by Microsoft (Also known as the Enron of Norway...) That one was painful to live through, we got forced to migrate to Windows and everything went sideways. That was almost 10 years ago with quite a big cluster (tens of nodes).

Don't forget Powerset!

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#146

Post by nova22033 »

SOLR is great but it's a pain to manage it in the cloud. If you lose an EC2 instance, there is manual work involved when you bring up a new instance. You have to tell the new instance servers which shards they're going to replicate. If the EC2 instance hosting shard1 replica2 goes down, you can't just bring up a new instance and have it be replica2. You need to use the API(which is just a call to a bunch of URLs) to get the new instance to be part of shard1. Also, a good cloud overview UI would be nice. 8.1.1 does have some improvements.

Also, SOLR speed is almost directly proportional to disk speed. If you index is on solid state drives with high iops, you'll be fine.

Backing up a large index is a little painful too.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#147

Post by jerska »

lm28469 wrote: post #6

Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.

I’ll just chime in to say that Algolia runs on an home-made C++ engine. [I work at Algolia]

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#148

Post by __blockcipher__ »

ragle wrote: post #79
busterarm wrote: post #46

Earlier quoted context omitted.

> Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. There's a 7-figure/yr Elastic Cloud customer I work w…

> AWS's hosted solution is poor What has been poor about it in your experience?

Not the GP, but you aren't allowed to touch settings like the shard recovery rate.

If a configuration change (changing # of nodes, instance type, etc) goes wrong, your cluster indefinitely gets stuck in Processing due to a race condition. The only way to get unstuck is to file a ticket. The company I'm at doesn't pay for AWS support, so at one point we ended up completely tearing down our cluster and rebuilding a new one (via Terraform) after getting tired of waiting for the reps. (They advised us to cut off log flow to let the system get out of processing, which we did, but it didn't work because once it gets stuck in processing like that it's just completely stuck).

It's difficult to troubleshoot issues - you can get some logs via Cloudwatch, but they're hard to search through and I'm not entirely positive everything shows up there.

Amazon is always several releases behind Elasticsearch versions.

--

Elastic.co's offering looks much better, just by reading their excellent comparison article: https://www.elastic.co/blog/hosted-elasticsearch-services-ro...

(We haven't used Elastic.co but what they say makes sense and I imagine their service is much better)

--

Once you hit a big enough scale - for us, we're pushing about 2TB a day of logs (that's before accounting for replication of course), it doesn't make sense to stay on Amazon's hosted service.

I'm in the process of advocating for in-housing our Elasticsearch setup and just building on top of ec2. Elasticsearh seems like the perfect candidate for Kubernetes since rebalancing is automatic and the affinity rules are simple (every Elasticsearch instance needs its own node). Cluster autoscaling (i.e. node-level, not _horizontal_) just makes too much sense.

Unfortunately I haven't gotten the go-ahead to take it inhouse, but I've been gunning for the project for some time now, so I'm hopeful I'll get the opportunity.

--

BTW, totally unrelated but for anyone managing Elasticsearch, make sure you have your shard count tuned properly. When I came to this company, they had their data way oversharded with primary shards varying between hundreds of kb to a few gb; i.e. orders of magnitude difference. Switching to ~50 GB shards (done via simplifying the way we were indexing) massively improved performance.

Also i3 instances > anything with EBS.

[/ramble]

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#149

Post by mlthoughts2018 »

I work professionally in this space and I can say over the past ~10 years, my full time job at 4 companies has been almost entirely to migrate away from SOLR / Lucene solutions and implement custom in-house search indexes.

Most of the time it has been for performance reasons. SOLR / Lucene have very poor performance characteristics, especially when needing to support custom sort ordering and heavy use of filters.

On other occasions it has been because you can’t easily extend search indexes to more advanced use cases, like similarity-based reverse item search, collaborative filtering, more advanced treatment of cold start issues / bias towards existing popular content / trending search.

A lot of small or medium sized companies naively figure they’ll just use SOLR etc to get something out of the box, or for side channel problems that are smaller scale.

But you come to regret it pretty fast because you end up needing one standardized way to build and deploy search indices and it has to support all the bells and whistles that SOLR can’t _and_ be faster than SOLR, for the big product use cases.

A lot of product companies now are beginning to use word-vector approaches with nearest neighbor libraries like ANNOY, as the first solution instead of the solution you eventually have to migrate to when you realize SOLR does not actually support your use case, not even as a means to get it up & running quickly.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#150

Post by shepmaster »

crawdog wrote: post #27

Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…

Vivisimo was acquired by IBM.

Source: I have the capital V from the building in my house.

(I see now atambo has already noted this typo)

Post reply on HN