Once upon a time, there was a search product and one of the data sources that it could search was a Solr/Lucene database. This should be no problem, since search is what Solr does. It should be as simple as passing the user's query through to Solr and then reading the response. The problem was, it was important to know exactly which parts of any matched records were relevant to the search.
The Guy Before Me™ decided that the best way to implement this would be to split the user's search into individual words, perform a separate search query through Solr's HTTP API for each individual word, and then do a bunch of very clever and complex post-processing on the result sets to combine them into a single set of results.
This led to endless headaches due to horrible performance. Imagine if you wanted to implement web search this way. How would you synthesize the results for the search "boston plumbers" given the search results for "boston" and the search results for "plumbers?" You would need tens of thousands of results for each search term to find even one match that applies to both terms. Now scale this to getting hundreds of results to present to the user. Now scale this to n search terms.
I was tasked with making this take less than 8,000ms for a simple query. I spent a while getting to understand how this code worked and building out performance tests so that we could determine how it would behave under load (we didn't have any users yet). The results were pretty grim. I presented two possible options for moving forward:
1. Move this crazy result-set-intersection logic closer to the data. I could build a custom Solr plugin to do this stuff inside the Solr server so that we didn't need to copy gigantic result sets across the network from Solr to the application server for every query.
2. Delete ALL of this nonsense because literally exactly what this whole mess of code was meant to accomplish is already implemented in Solr. They call it highlighting. It's one of the marquee features of the program. I can't stress enough that this is precisely, perfectly, unequivocally, the exact thing that all of this complexity was meant to accomplish.
My manager thought it would be a shame to throw away all of that very expensive code and lose the flexibility of an in-house solution. So we went with option one. I spent the next month writing a Solr plugin that reproduced the original logic. It was still slow as mud so I sharded the data across multiple Lucene servers and distributed the algorithm across them with a map/reduce sort of scheme.
In the end, it all worked great. It was fully ten times slower than the solution already built into Solr, but it worked.
The startup later ran out of runway trying to build a big-data-sized in-memory distributed database from scratch to speed up search. The founder (also the lead developer while I was there) insisted that everyone use raw C-style arrays and a custom in-house hash table implementation because he thought STL was too slow. Basically, "not invented here" was in the DNA of that company. I'm surprised we even used commodity hardware and didn't design some kind of in-house search coprocessor that would do everything in silicon.