Yep, most of the audience facing search functionality across our sites (https://www.bbc.co.uk) is powered by various Solr clusters, hosted on-prem and cloud.
Ask HN: Are Lucene/Solr/ES Still Used for Search?
31–40 of 223 posts
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#32Search gets less attention these days, but mostly because the current tools work so well. Scaling Elastic is still a bit of a dark art, but our company likely wouldn't exist without lucene/elastic. They take a bit to learn and use correctly, but they are incredibly powerful.
What do you mean by saying scaling Elastic is a dark art? Care to expand on that?
I'm by no means heavily experienced in this, but my current employer has a big demand for this. (e-commerce)
First there's the Docker + Kubernetes architecture that ES lends itself to really well. Then (depending on your use-case) there are concerns like hot/warm architecture, node types, ETL/indexing processes. ES recently moved over to openJDK, so there's a couple intricacies there (i.e. JVM heap size)
Then, there's document/query structure. In no particular order:
- Do you have any parent/child relationships?
- Do you have stop-word lists developed?
- Can search templates help your queries?
- How will you interface with ES? It has REST APIs, but it's recommended to not expose ES directly to your applications.
- Some advanced querying possibilities like customizing tokenizers, normalizers, and a bit of internationalization.
- Oh, we haven't even discussed security yet.
- Also, ES isn't meant to be a primary data storage. This is more so a "cache", but not quite like Redis. So, you'll need a DB elsewhere most of the time.
All of this changes depending on if you're using it for SIEM, e-commerce, AI/ML, etc. Also, Elastic now provides their own SIEM solution, a pre-built search solution (AppSearch + Search UI), built-in security features. Check out the new ES 7.2 update; it's kinda nuts.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#33Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.
There's folks working on Bleve (written in Go) and developers that I work with want to use it (we use Elasticsearch heavily), but as I've told everyone like you just did, Lucene has a 20 year head start.
Thing is, there's heavy demand for something more performant than Elasticsearch, so eventually the market will provide.
Meanwhile, Redis Enterprise is trying to grab some 'share with RediSearch, which has some severe caveats IMO that make it not a great fit for most.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#34Yeah, I work pretty much daily with Solr and related stuff (PHP, JSON). Still used quite a lot in PHP and Drupal scene.
Yup. Lots of Drupal sites use Solr. There are very good contributed modules that make using Solr with Drupal a doddle.
Same here - using SOLR with Drupal and it's pretty simple and effective
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#35I use Solr & ElasticSearch heavily — they're “boring” in the sense that they do a lot of heavy lifting without many surprises and they scale easily into at least terabyte-sized indexes.
One area where this might be less true is that the full-text search in Postgres & MySQL have matured to the point where some basic applications might reasonably decide that it's not worth using a separate service.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#36We have been using https://www.algolia.com/ completely as a replacement for ES. Pros: - Managed search engine - Great API / Developer experience Cons: - Cloud only makes it hard for local development - Expensive (I guess it depends on the usage)
Worth noting this is what hacker news itself uses
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#37AFAIK, Reddit, Slack, Dice, Bloomberg, IBM, Apple all use Solr. Jira and Confluence use Lucene. Others use Elasticsearch and Fusion (commercial product on top of Solr).
See, for example: https://www.activate-conf.com/more-events for presentations from several past years on who and how uses Lucene/Solr.
Also, the new trend in jobs is "Relevancy Engineering", which is less about just setting up search engines and more on actually tuning them. That's where also Machine Learning and other AI techniques come in (Learning to Rank, Named Entity Recognition, sentiment analysis, etc.). Which was recognized by rebranding of the conference from Lucene/Solr revolution to Activate last year.
See also Haystack conference which focuses very specifically on relevance regardless of the specific search engine: https://haystackconf.com/
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#38The technologies are still heavily used but there might be slightly less demand for the skill set because:
1) Cloud search service - You are less likely to deal with setting up your own instance and concerns that go with it (sharding, etc) because most Cloud providers offer either ElasticSearch as a service or some form of turnkey deployment. You still have to do some low-level work like score manipulation but you don't deal with as much administration.
2) Search as a Service - https://en.wikipedia.org/wiki/Search_as_a_service. There are several companies that provide a Search SaaS offering. Typically they provide value adds above just running your ES service. Often they will provide web crawlers so you can just point them to your domain or they might provide other datasource integrations like pulling content from a database. You get access to Solr/ES functionality if you want it but you can get search running without going to that level if desired.
Either way a Lucene based stack is still in use.
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#39Very much so. For retail/catalog search SOLR dominates. There's a lot more customization available for relevancy/ranking OOB than Elastic. Drawbacks are managing indexing - SOLR cloud is much harder to manage. For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support. Lucene is the core engine behind both of these solutions. For fun…
There are no contacts in your profile (for pinging offline)
Re: Ask HN: Are Lucene/Solr/ES Still Used for Search?
#40Wikimedia uses ES and you can download their entire index for any of their sites wikipedia/travel/quotes etc.