Are there any objective metrics/ benchmarks that people test harnesses by? There are just so many now that it's hard to personally test them all or just trust the vibes.
Don't use any public benchmarks, every single one is worthless for your own use cases essentially.
Spend a day or two going through your existing chat sessions, and create your own private benchmark with test cases based on real tasks, that you don't share with anyone nor publicly. Make it easy to add/remove new harnesses and model combinations, make it give you a final score, ideally avoid using other LLMs for scoring, then use this to figure out if the new model/harness actually improves things for you.
I've been doing this for some time, and while most new releases show big increases in the benchmarks/evaluations, my own benchmark usually barely moves.