I build search engines for a living. While I appreciate the hacker spirit of your project with Solr, I also see this as a huge problem that leads to bad search experiences. Tuning boosts in Solr is not even close to a reasonable way to solve problems like this. Arguably not even for an underfunded library, but certainly not for a high web traffic consumer website.
For one, you need disciplined acceptance criteria in the form of both qualitative standards (things a non-technical manager can look at and say yes or no) as well as various relevancy measurements like mean reciprocal rank and normalized discounted cumulative gain (via acquiring human annotated data if needed).
When people only focus on qualitative feedback on top of boosts and hacks in an off the shelf tool, they usually end up with some weird witches’ brew of bizarre boosts and time-decay weighting that is extremely fragile and can’t be robustly changed or even understood without the qualitative performance going haywire. You need disciplined study of quantitative ranking metrics to know the drivers of performance, fall off as you move down the ranking position, and to make search index updates reproducible and make incremental improvement measurable.
Meanwhile if you only focus on quantitative metrics, you might miss obvious red flags. The relevance score used for NDCG might be biased some way. You might surface highly relevant results to only one context or sense of the words in a query (like only showing fruit for “apple” and never tech gadgets). You need people who make the subjective appraisal of quality for users to be looped in.
Here’s the point. When this is all missing, you will lose credibility with people making the decisions. They’ll hear some engineer babble about NDCG but then say the darn thing doesn’t work in the QA testing. Or they’ll say the qualitative results look OK and get angry when weird counter-examples pop up in the second or third results pages, which might have been measured with quantitative metrics.
When this happens, executives and managers just want to punt. They want the “nobody ever got fired for buying IBM” equivalent for search, and that’s how you end up with Confluence still only supporting exact title matching and having no ability for actual content relevancy search.
In this sense, the little projects showing “look what us non-specialists could cook up by hacking some boosts in Solr!” do a lot of harm and should not be considered as the plucky success stories they are often painted to be.