21–30 of 223 posts
I'm using Solr for https://www.findlectures.com , but I think Vespa looks interesting - lets you store feature vectors in the index, so you can do neat things to incorporate ML algorithms in ranking.
http://highscalability.com/blog/2013/1/28/duckduckgo-archite...
Search gets less attention these days, but mostly because the current tools work so well. Scaling Elastic is still a bit of a dark art, but our company likely wouldn't exist without lucene/elastic. They take a bit to learn and use correctly, but they are incredibly powerful.
For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support.
Lucene is the core engine behind both of these solutions.
For fun, lets look at the large Enterprise acquisitions over the years:
* Verity - bought by Autonomy
* Fast - bought by Microsoft (Also known as the Enron of Norway...)
* Autonomy - bought by HP (Look at the backstory on this deal!)
* Endeca - bought by Oracle
* Vivisimo - bought by Oracle
* Google - GSA (now Google Cloud Search, hosted solution)
Next, follow the path of online acquisitions:
* IndexTank - bought by LinkedIn
* Swiftype - bought by Elastic
There's a number of interesting independent players still. Coveo plays in the Enterprise space, but it's a hard market. Algolia is doing great in the commodity online search space and seems to be growing well.
This is an area I think is open to more competition. Especially with AI/ML technologies available around Document Understanding - the Enterprise market is open for a good on-prem upstart to really take off.
Ping me offline if you have additional questions - spent almost 20 years in the space and ran a search company of my own.
While I have not used Solr recently, it has evolved along with ES as a solid product with a solid community. Nothing against it; it's a solid choice and there are probably people offering to host that as well. Either way you are using Lucene. Using Lucene directly makes no sense unless you know what you are doing and have a need to do that. If you have to ask that means it's not for you.
If search is important to your use case and things like relevance, precision, and recall have real impact on your business, you should get some specialists involved and not reinvent the wheel or make all the rookie mistakes. Somebody like me basically ;-). However, that can be expensive and if search is not that critical, just sign up for one of the search as a plug in solution type products out there and don't bother running a lot of infrastructure. E.g. Elastic offers a thing called App Search and it probably covers most of the simple needs and is stupidly easy to get started with. There are several competing products probably that I can't vouch for.
You can always upgrade to something proper later. Things like mongo and postgres also have some limited capabilities here and you can get away with doing some simplistic stuff with sql even. However, there's a point where you hit a brick wall and there's something you need that is simply hard/impossible using that or where you end up reinventing a lot of stuff that things like Elasticsearch probably do better.
For small projects (for some 1000s of documents), I'd probably go with Postgresql FTS if possible. Sphinx/Solr for anything with indices smaller than a couple 100GBs After that, ES seems reasonable & worth the overhead
EDIT: my biggest issue with ES is that it seems to be specifically engineered to sell you support. So get a managed version if you can.
Afaik almost everything runs Lucene under the hood, it's 20 years old, no one is going to build something as good any time soon. I suppose some company like Google have their own in house solution but otherwise it'll always be something built on top of Lucene. I guess you don't see much demand because for a lot of use cases the basic setups are good enough.
About seven years ago, I got a contracting gig for a website that wanted a "search engine". I remember thinking "Solr/Lucene is old, not pure-functional, and therefore awful!" and decided to build my own. Somehow I even managed to convince the client that this was a good idea. I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was s…
TLDR: Lucene Delivering a conference talk [1] later this year about it.
[0]: https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-...
[1]: https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-531...