Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Are Lucene/Solr/ES Still Used for Search?

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 223 posts

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#161

Post by rpedela »

porker wrote: post #118

This has been a great thread, and there's some heavyweight indexes here. But what about at the other end of the scale? Say when you've got 10k-50k contact details (name, email, phone) and you want to provide a quick, autocomplete lookup. I've used basic SQL string matching for this, but it doesn't catch mis-spellings and the rest. Running SOLR or ES is overkill for this. Is there a tool that fits this niche?

What's wrong with running Solr/ES? It is trivial to run either in standalone mode, and it is a lot easier to set up autocomplete with misspelling support than messing with PG. Algolia is a good option if you have the budget.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#162

Post by adventured »

I'm still using an older version of Sphinx. I love it. It's fast, moderately flexible, very lightweight, easy to set up and produces good enough results. I have also found it to be highly reliable (at least the version I've been using for years). It's not useful for anything that needs hyper scale (Twitter et al), however for the next tier of scale below that it generally does well if you know how to leverage its strengths.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#163

Post by adventured »

altendo wrote: post #78

Are people still using Sphinx Search ( http://sphinxsearch.com ) at all? It doesn't seem like it gets many releases anymore...since they unpublished the source code, it's hard to see how much activity there is.

I'm using a 2.x version of Sphinx and have been for many years. I refuse to upgrade the version. I haven't been able to break/crash the version I'm using under nearly any common circumstances or loads, so I'm sticking with it until I find an alternative that is dramatically better. I've learned every nuance of it over time and can make it sing & dance exactly how I want it to. I consider it a spectacular piece of software; it does a thing and does it well, reliably.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#164

Post by aldoushuxley001 »

pmarreck wrote: post #142

Two years ago I decided to go with Postgres' built-in fulltext search instead of adding another dependency like ElasticSearch, and I believe I've profited from that in much less maintenance while still getting quite good performance/features.

Any tips for scaling Postgres-only fulltext search?

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#165

Post by aldoushuxley001 »

riku_iki wrote: post #140
holstvoogd wrote: post #29

yeah, there aren't many alternatives unfortunately. I've used Sphinx a lot, but am now stuck with ES and it is horrible to operate, probably because we don't need a cluster solution so it is total overkill. Yeeeah for technical debt. For small projects (for some 1000s of documents), I'd probably go with Postgresql FTS if possible. Sphinx/Solr for anything with indices smaller than a couple 100GBs After that, ES seems…

> I'd probably go with Postgresql FTS if possible. Sphinx/Solr for anything with indices smaller than a couple 100GBs After that, ES seems reasonable & worth the overhead why do you think you can't throw 1TB of data on postgresql?

Could postgres FTS handle millions of documents within a reasonable timeframe?

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#166

Post by hiram112 »

tombert wrote: post #17
lm28469 wrote: post #6

Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.

About seven years ago, I got a contracting gig for a website that wanted a "search engine". I remember thinking "Solr/Lucene is old, not pure-functional, and therefore awful!" and decided to build my own. Somehow I even managed to convince the client that this was a good idea. I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was s…

Can you explain a bit why ES isn't a good solution for storing data itself?

I inherited a legacy Mongo solution, and all the data is duplicated and indexed in ES, so I've always wondered why we're using both. Mongo has none of the SQL capabilities that would make my life easier, and the types of queries allowed by Mongo could be done with ES.

What are the negatives of ES alone?

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#168

Post by manigandham »

sandGorgon wrote: post #155
otisg wrote: post #127

Earlier quoted context omitted.

> so tired of Elastic just randomly killing their clusters out of nowhere and having to spend basically triple to deal with it that they're bringing it all in house. That is because Elastic Cloud is not a fully managed Elasticsearch. People often don't get that with Elastic Cloud you are still responsible for your ES cluster. That's one of the differentiators that e.g. Sematext has (disclaimer: founder). > AWS's host…

Why are all ELK stack based logging solutions so much more expensive than custom rolled solutions like logdna, datadog, papertrail,etc. The per gb cost on the other ones start at 1.20 $/gb and goes till 2$/gb. While almost all hosted ELK solutions start off at 3$/gb. Im asking because i would very much like to adopt an ELK based hosted solution..but I'm not able to justify paying double. Is it that running+resource c…

Yes it's mostly about resource costs. ES is a generic search that can handle logs but isn't focused on it, and indexing everything can be costly. There are major efficiencies gained in creating your own log-focused storage and access methods with object storage, columnar formats, and zone mapping, etc.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#169

Post by manigandham »

hiram112 wrote: post #166
tombert wrote: post #17

Earlier quoted context omitted.

About seven years ago, I got a contracting gig for a website that wanted a "search engine". I remember thinking "Solr/Lucene is old, not pure-functional, and therefore awful!" and decided to build my own. Somehow I even managed to convince the client that this was a good idea. I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was s…

Can you explain a bit why ES isn't a good solution for storing data itself? I inherited a legacy Mongo solution, and all the data is duplicated and indexed in ES, so I've always wondered why we're using both. Mongo has none of the SQL capabilities that would make my life easier, and the types of queries allowed by Mongo could be done with ES. What are the negatives of ES alone?

It's not reliable: https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my...

The v7 upgrade to a new cluster protocol (zen2) has improved things but overall the system has a long history of losing or destroying data. It's better to have a primary OLTP system that's ACID and reliable while using ES as the secondary search source. You can also remove the _source field if you just need matches without the original content.

It's common to see pattern used with a relational database since, as you can see, Mongo doesn't buy you much else as another document-store.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#170

Post by manigandham »

Elasticsearch is very popular because it works well for generic searching and can be customized for lots of unique scenarios. There's competition on the infrastructure side using something other than Java/JVM though:

For Rust, there's Toshi: https://github.com/toshi-search/Toshi which is built on top of Tantivy: https://github.com/tantivy-search/tantivy

For C++, there's Xapiland: https://github.com/Kronuz/Xapiand

For Go, there's Blast: https://github.com/mosuka/blast built on Bleve: https://github.com/blevesearch/bleve

Post reply on HN