Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Are Lucene/Solr/ES Still Used for Search?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 223 posts

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#171

Post by manigandham »

Izkata wrote: post #137
ps101 wrote: post #108

Earlier quoted context omitted.

Can you elaborate on the reasons for this?

Coming from experience with Solr, the answer is far simpler than the link in the sibling comment indicates: Data is processed and manipulated on import, and it's difficult - sometimes impossible, depending on the field config - to get that data back out in the format it was imported. Any changes to the schema, such as switching fields between indexed/not indexed/stored/not stored, requires reimporting the data to pop…

Elasticsearch has a _source field [1] that stores the entire original document and is enabled by default. It's required to support features like highlighting in results. ES also has a reindex API that specifically makes use of this [2].

1. https://www.elastic.co/guide/en/elasticsearch/reference/curr...

2. https://www.elastic.co/guide/en/elasticsearch/reference/curr...

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#172

Post by bryanrasmussen »

rickmode wrote: post #62
bryanrasmussen wrote: post #52

Earlier quoted context omitted.

I think the instructor's response in the linked article is a reasonable defense, you don't really know that data is deleted all the way down to the file level. It is just marked as deleted and could be retrieved by someone clever enough to do so. At some point in the future it will be really deleted. I don't think the GDPR regulatory agencies are operating at a technical level that they would make an argument that it…

When using ES for indexing and not the primary store, you can (and should) periodically fully reindex the data set. You can use a blue / green pattern — create a new index then swap from the old one to the new one. ES supports aliases, making this swapping transparent to the apps using the index. Now you have more options. If it is easy to delete specific users from the primary database, the deleted users will natura…

thanks, I wasn't arguing that using ES as primary was good. Just don't necessarily see the GDPR argument as being a reasonable one. Although I've seen some startups using Mongo as primary and have to wonder if there would be that big a difference in using ES at that point (not a Mongo dig as I've kept away from it for various reasons)

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#173

Post by porker »

rpedela wrote: post #161
porker wrote: post #118

This has been a great thread, and there's some heavyweight indexes here. But what about at the other end of the scale? Say when you've got 10k-50k contact details (name, email, phone) and you want to provide a quick, autocomplete lookup. I've used basic SQL string matching for this, but it doesn't catch mis-spellings and the rest. Running SOLR or ES is overkill for this. Is there a tool that fits this niche?

What's wrong with running Solr/ES? It is trivial to run either in standalone mode, and it is a lot easier to set up autocomplete with misspelling support than messing with PG. Algolia is a good option if you have the budget.

> What's wrong with running Solr/ES?

With this small quantity of data, usually the app's running on a small VM. I'm wary of running anything Java, having had it require large amounts of RAM before.

That said, I haven't touched JVM stuff for 5+ years.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#174

Post by softliving »

arafalov wrote: post #37

AFAIK, Reddit, Slack, Dice, Bloomberg, IBM, Apple all use Solr. Jira and Confluence use Lucene. Others use Elasticsearch and Fusion (commercial product on top of Solr). See, for example: https://www.activate-conf.com/more-events for presentations from several past years on who and how uses Lucene/Solr. Also, the new trend in jobs is "Relevancy Engineering", which is less about just setting up search engines and more…

Bloomberg, Apple, IBM also use Elasticsearch or more Elastic stack!

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#175

Post by softliving »

ksec wrote: post #105
lm28469 wrote: post #6

Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.

>Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. Vespa [1] would like to have a word with you. [1] https://vespa.ai

move on! IMO donated tool, Yahoo can't maintain it, give it to community reduce the cost. If only it was a new project which wanted to address search in a different way.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#176

Post by ksec »

softliving wrote: post #175
ksec wrote: post #105

Earlier quoted context omitted.

>Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. Vespa [1] would like to have a word with you. [1] https://vespa.ai

move on! IMO donated tool, Yahoo can't maintain it, give it to community reduce the cost. If only it was a new project which wanted to address search in a different way.

If anything Vespa has actually gotten faster in development since it Open Sourced.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#177

Post by ksec »

bratao wrote: post #134
otisg wrote: post #128

Earlier quoted context omitted.

That's interesting to hear. Would love to read about how Vespa compares to Solr and ES. This may be of interest to you: https://sematext.com/opensee/report/project/trend?q=ElasticS... Would you happen to know how Vespa compares to ES in terms of memory or CPU footprint? Have you done apples to apples comparison by any chance?

I do not have a completely fair comparison. But a migration from Elasticsearch 5 (2016) to the Vespa 7 (2019) we reduced half of our nodes, and cut in half the average response time. Another amazing feature during the migration, is that Vespa allows you to reduce or increase the number of nodes dynamically. And it take full care about the data distribution. In ES we (used to) had to follow the limits of the pre-confi…

I have been wondering why Vespa isn't getting much traction. Everyone still defaults to ES, even in new project.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#178

Post by ngrilly »

pmarreck wrote: post #142

Two years ago I decided to go with Postgres' built-in fulltext search instead of adding another dependency like ElasticSearch, and I believe I've profited from that in much less maintenance while still getting quite good performance/features.

Do you use ts_rank? PostgreSQL FTS is very efficient until you want to rank the results according to their relevance. This is because the data necessary to the ranking are not in the GIN or GIST index. They are in the heap, and this triggers a lot of random IOs.

Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?

#179

Post by glintik »

joking wrote: post #81
srameshc wrote: post #60

Search built using Postgres is underrated. It can do a lot if used properly.

there are on different levels, a search on a sql database will have real time results, while on solr / elasticsearch it's going to have a delay (from milliseconds to minutes). That delay gives the advantage to build a series of data structures much more suited for search than the ones on a database. I built several search systems for classified listing sites, something like solr is a life saver once you get enough tr…

ES search have no any delay, if data is really commited to index. That's also true for any general DB - Postgres etc.
Post reply on HN