> “But it works?”
This is the big red flag, when non-specialists hacking on Solr boosts are claiming something works because of a few qualitative test cases.
“It works” is a statement that only applies after you’ve done qualitative and quantitative goodness of fit testing.
You wouldn’t have a random IT employee make a stock-trading algorithm and then test it on a month of data and call it a success.
For a search solution to “work,” it needs to pass quantitative and qualitative checking, and be explainable to stakeholders and be reproducible / incrementally updateable. The training and arrival at hyperparameters all need to be reproducible and based on the outcome criteria they are meant to solve.
Making some hacks into a demo that superficially looks good is not at all the same as “it works.”