Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Are Lucene/Solr/ES Still Used for Search?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 223 posts

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#131

Post by mish15 »

blyry wrote: post #107

We use elasticsearch to power our ecommerce search, and it works pretty well, but we're considering moving to a commercial product, or solr, to get closer to personalized results based on our knowledge about the user. We just rewrote our internal search API from a windows service indexer with lucene indices and a vb.net SOAP api in iis to a netcore service, hosted in k8s, that splits out ingest, analysis, storage and…

We have an early access product for personalized ecommerce search @Sajari if you are interested. One early access company is on track to generate $30 million in additional revenue from switching (over 10% search conversion increase). That is across millions of skus and hundreds of products updated per second also.

We are also looking at releasing this as a k8s deployable product. It's all k8s services and gRPC already...

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#132

Post by jzawodn »

pQd wrote: post #117
altendo wrote: post #78

Are people still using Sphinx Search ( http://sphinxsearch.com ) at all? It doesn't seem like it gets many releases anymore...since they unpublished the source code, it's hard to see how much activity there is.

https://manticoresearch.com/ is the lively, open source, fork of Sphinxsearch. that's where some of the earlier developers from the project moved to. it's used as a text-search backend on craigslist.

Definitely. I love me some manticore. :-)

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#133

Post by jzawodn »

drenvuk wrote: post #103
makillik wrote: post #85

Earlier quoted context omitted.

We just migrated from Sphinx to using the full text search indexes in PostgreSQL, we had to deal with some changes in how special characters are handled, but it's worked well enough.

Do you have any numbers on your requests or searches per second in a real use case? I've really been wondering this as I've been considering manticore which is the major sphinx fork.

If you need high query rates, I suspect manticore will stand up quite nicely.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#134

Post by bratao »

otisg wrote: post #128
bratao wrote: post #116

Earlier quoted context omitted.

You can contact me. I'm also planning to do a blog post comparing with Solr and Elasticsearch. I think that naturally it takes some time to adopt a solution like that. And the ecosystem it still at it infancy. But, randomly, a project using Vespa appeared in my GitHub timeline today ( https://github.com/rdoume/News_API ). So, the adoption is increasing. For me Vespa is a absolute game-change, in features and as someo…

That's interesting to hear. Would love to read about how Vespa compares to Solr and ES. This may be of interest to you: https://sematext.com/opensee/report/project/trend?q=ElasticS... Would you happen to know how Vespa compares to ES in terms of memory or CPU footprint? Have you done apples to apples comparison by any chance?

I do not have a completely fair comparison. But a migration from Elasticsearch 5 (2016) to the Vespa 7 (2019) we reduced half of our nodes, and cut in half the average response time. Another amazing feature during the migration, is that Vespa allows you to reduce or increase the number of nodes dynamically. And it take full care about the data distribution. In ES we (used to) had to follow the limits of the pre-configured number of shards/replicas during the Index creation.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#135

Post by boyter »

ragle wrote: post #79
busterarm wrote: post #46

Earlier quoted context omitted.

> Elasticsearch, definitely. I always recommend using it in hosted form and not running your own cluster. That allows you to focus on getting data in and out of your cluster instead of sinking time into doing devops. I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive. There's a 7-figure/yr Elastic Cloud customer I work w…

> AWS's hosted solution is poor What has been poor about it in your experience?

Not OP but the list I had from a year ago, didn’t support all multi az (only 2), lack of admin features meant it broke and you could work out what happened, limited choice of instance type, the backup happened once a day regardless if you needed it and it had issues under load.

The backup issue has been resolved I believe but the others...

On AWS I’d suggest elastic on ECS and you can use the leftover compute on the cluster to run other applications effectively for free.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#137

Post by Izkata »

ps101 wrote: post #108
parthdesai wrote: post #20

Earlier quoted context omitted.

>The SQL server is always going to be our source of truth, probably. ES is not supposed to be your source of truth.

Can you elaborate on the reasons for this?

Coming from experience with Solr, the answer is far simpler than the link in the sibling comment indicates: Data is processed and manipulated on import, and it's difficult - sometimes impossible, depending on the field config - to get that data back out in the format it was imported.

Any changes to the schema, such as switching fields between indexed/not indexed/stored/not stored, requires reimporting the data to populate those fields, data which you're not likely to have if it was your primary store.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#138

Post by mosen »

janemanos wrote: post #136

I guess more and more people are turning towards search engines directly integrated into databases, like ArangoSearch of ArangoDB to combine search with other needs https://www.arangodb.com/why-arangodb/full-text-search-engin...

I assume you work for ArangoDB?

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#140

Post by riku_iki »

holstvoogd wrote: post #29

yeah, there aren't many alternatives unfortunately. I've used Sphinx a lot, but am now stuck with ES and it is horrible to operate, probably because we don't need a cluster solution so it is total overkill. Yeeeah for technical debt. For small projects (for some 1000s of documents), I'd probably go with Postgresql FTS if possible. Sphinx/Solr for anything with indices smaller than a couple 100GBs After that, ES seems…

> I'd probably go with Postgresql FTS if possible. Sphinx/Solr for anything with indices smaller than a couple 100GBs After that, ES seems reasonable & worth the overhead

why do you think you can't throw 1TB of data on postgresql?

Post reply on HN