Ask HN: Are Lucene/Solr/ES Still Used for Search? • Hacker News Board

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#191

Post by peterdm » Sat, Jul 20, 2019, 7:58 PM UTC

arafalov wrote: post #37

AFAIK, Reddit, Slack, Dice, Bloomberg, IBM, Apple all use Solr. Jira and Confluence use Lucene. Others use Elasticsearch and Fusion (commercial product on top of Solr). See, for example: https://www.activate-conf.com/more-events for presentations from several past years on who and how uses Lucene/Solr. Also, the new trend in jobs is "Relevancy Engineering", which is less about just setting up search engines and more…

Until about 2014-2015, many large companies wouldn't have looked twice at Elasticsearch. The companies you list using Solr have been invested in search for 10 years or longer (predating Elasticsearch), and may have high switching costs.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#192

Post by manigandham » Sat, Jul 20, 2019, 9:32 PM UTC

aldoushuxley001 wrote: post #165

riku_iki wrote: post #140

Earlier quoted context omitted.

> I'd probably go with Postgresql FTS if possible. Sphinx/Solr for anything with indices smaller than a couple 100GBs After that, ES seems reasonable & worth the overhead why do you think you can't throw 1TB of data on postgresql?

Could postgres FTS handle millions of documents within a reasonable timeframe?

Yes, it's searching against the ts_vector data types which can be indexed.

The problem with PG FTS is that it doesn't have advanced search functionalities (fuzzy matching, faceting, term distance, highlighting results) and it lacks the modern relevance scoring systems so that'll be the limiting factor instead of speed.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#193

Post by manigandham » Sat, Jul 20, 2019, 9:40 PM UTC

manigandham wrote: post #169

hiram112 wrote: post #166

Earlier quoted context omitted.

Can you explain a bit why ES isn't a good solution for storing data itself? I inherited a legacy Mongo solution, and all the data is duplicated and indexed in ES, so I've always wondered why we're using both. Mongo has none of the SQL capabilities that would make my life easier, and the types of queries allowed by Mongo could be done with ES. What are the negatives of ES alone?

It's not reliable: https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my... The v7 upgrade to a new cluster protocol (zen2) has improved things but overall the system has a long history of losing or destroying data. It's better to have a primary OLTP system that's ACID and reliable while using ES as the secondary search source. You can also remove the _source field if you just need matches without the original…

Follow up: MongoDB is adding full-text search capabilities: https://www.youtube.com/watch?v=4QUGWnz-XaA

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#194

Post by quickthrower2 » Sat, Jul 20, 2019, 9:57 PM UTC

ES powers search one of my side projects https://dealscombined.com.au.

The ability to not only full text search but to do it fast, to tune the lexical behaviour (lowercase, plurals, stemming etc.) and to top it all combine geo search pretty much left any other solution in the dust. I even considered some paid solutions.

I also considered postgres which looked strong but I felt it’s be harder to set up these features and that the full text would be weaker although geo might be stronger but my geo needs are simple.

ES was easy to set up to do this, taking about 2 hours of tuning. I used AWS so I didn’t have to figure out how to install it. I admit I had a mental model of ES from ELK-ing at work.

At some point when the site gets more traffic I’ll tune the search so that rather than nearest matching, I’ll score bit the distance and the words and order by perceived relevance. Ie weigh up both how close something is with how well the words match.

ES is a pretty amazing tech and it’s the easiest way to set up a decent quality free text search for your site.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#195

Post by ngrilly » Sat, Jul 20, 2019, 9:58 PM UTC

aldoushuxley001 wrote: post #164

pmarreck wrote: post #142

Two years ago I decided to go with Postgres' built-in fulltext search instead of adding another dependency like ElasticSearch, and I believe I've profited from that in much less maintenance while still getting quite good performance/features.

Any tips for scaling Postgres-only fulltext search?

https://github.com/postgrespro/rum

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#196

Post by yehosef » Sun, Jul 21, 2019, 12:16 AM UTC

vosper wrote: post #109

meddlin wrote: post #32

Earlier quoted context omitted.

I'm by no means heavily experienced in this, but my current employer has a big demand for this. (e-commerce) First there's the Docker + Kubernetes architecture that ES lends itself to really well. Then (depending on your use-case) there are concerns like hot/warm architecture, node types, ETL/indexing processes. ES recently moved over to openJDK, so there's a couple intricacies there (i.e. JVM heap size) Then, there'…

> ES recently moved over to openJDK, so there's a couple intricacies there (i.e. JVM heap size) My current employers uses ES - we're on 6.8, planning to move to 7 in a few months. Judging by the other replies here I'd say we have a reasonably large cluster (150+ i3.2xlarge instances, billions of documents), so tuning the cluster is very relevant to us. Could you expand on how things have changed with the move to Open…

>I've seen some claims online that, contrary to what Elastic recommends in their docs, a few machines with huge heaps (100+ gb) is the way to go, rather than many machines with 20gb heaps.

Usually the recommendation is less than 32GB - this link has some more discussion about it: From https://discuss.elastic.co/t/es-lucene-32gb-heap-myth-or-fac...

It seems whether it's better or worse depends on your data set . But I would love to see tests of different kinds of workloads with large or smaller heaps.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#197

Post by yehosef » Sun, Jul 21, 2019, 12:28 AM UTC

rchasman wrote: post #77

We use ElasticSearch at Lawmatics, and it powers more functionality than just our search! We use it to power our Automation targeting engine, reporting features, audience builder and pagination, filtering, sorting of data tables. We denormalize associated records into one Index. And any record that we need to find based on user-defined queries will go through ES since it's much simpler to metaprogram queries across d…

This is one of the undersung benefits of ES in my eyes. Relevancy results requires tuning of the indices and queries and in most cases (that non IR-experts would program) ES will give as good results and be as easy or easier to implement as Solr.

But after you've gotten over that, you realize that this new tool can do lots more things than just text search. Time series metrics, BI, predictive ML, APM, etc. with relatively little work. With Solr, you could do those non-IR tasks, but it's going to feel much more awkward, IMO.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#198

Post by peterdm » Sun, Jul 21, 2019, 2:18 AM UTC

garysieling wrote: post #15

I'm using Solr for https://www.findlectures.com , but I think Vespa looks interesting - lets you store feature vectors in the index, so you can do neat things to incorporate ML algorithms in ranking.

Feature vectors do tend to get incorporated in relevance tuning (regardless of the engine), but from what I've heard of Vespa, features (and ML in general) are first-class citizens, whereas with Elasticsearch and Solr, text statistics are your first-class citizens, and you're adding in additional features and integrating ML at the periphery.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#199

Post by peterdm » Sun, Jul 21, 2019, 2:34 AM UTC

blyry wrote: post #107

We use elasticsearch to power our ecommerce search, and it works pretty well, but we're considering moving to a commercial product, or solr, to get closer to personalized results based on our knowledge about the user. We just rewrote our internal search API from a windows service indexer with lucene indices and a vb.net SOAP api in iis to a netcore service, hosted in k8s, that splits out ingest, analysis, storage and…

For e-commerce search, personalization with Elasticsearch takes a similar level of effort as with Solr. Don't re-platform under the impression it will make personalization easier. It still takes data collection and experimentation but can be accomplished on Elasticsearch. Feel free to contact me if you have questions.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#200

Post by peterdm » Sun, Jul 21, 2019, 3:19 AM UTC

samsk wrote: post #124

porker wrote: post #118

This has been a great thread, and there's some heavyweight indexes here. But what about at the other end of the scale? Say when you've got 10k-50k contact details (name, email, phone) and you want to provide a quick, autocomplete lookup. I've used basic SQL string matching for this, but it doesn't catch mis-spellings and the rest. Running SOLR or ES is overkill for this. Is there a tool that fits this niche?

With Postgresql you can use pg_trgm, might be not as powerful as what SOLR/ES provides, but easier to run.

Upvoted. This is how Postgres supports "fuzzy searching" which helps with misspellings. https://www.rdegges.com/2013/easy-fuzzy-text-searching-with-...

Completion response-time will be slower than Solr, Elasticsearch, Algolia, etc... but if you're already running Postgres, this may be the fastest to deliver for you.