Everyone is building LLM routers, we deprecated ours

My understanding was that models can be post-trained to assess their own confidence on short prompts within a level of accuracy (there was an article about this a few weeks ago that I can't find), but can't use it to reason, and that research has shown that internally they effectively have a measurable sense of truth but can't surface it:

https://arxiv.org/abs/2410.02707

I mangled what I was trying to say but the point I guess is, it is effectively in there, and researchers can see it and therefore score it, but it is not something the LLM can use.

The problem seems to me (layman's understanding at best) that the LLM is inherently confident within the words of its answer, because of what the model is trained to do and how it is trained. So you can never get an accurate "I don't know this" from a model while it is answering; it is bullshitting.

(One of the things that is most interesting to me at the moment is asking a small local LLM what it knows about a topic and then testing it. It can have no idea of what it wasn't trained on, but any question about what it knows is seemingly going to trigger it to work through all the summary word associations it did find in training, so it can sort of summarise its knowledge that way, with some likelihood of success, without any sense of introspection)

The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut into lower levels of abstraction.

E.g. you should consider speculative decoding to be one (very conservative) form of routing and note that you can’t implement that for the labs from the outside.

What I mean is that most tasks can be recursively fragmented into smaller tasks, and once you've hit suitable leaf nodes -- where the task is very granular -- you can begin to deterministically show which models perform better or worse for that specific task. Then, when your agent is running a workflow, you may use various models for different steps in a workflow. For example, some models may excel at exploration, some at determining a good architectural fit for an implementation, some at actually writing the implementation, and so on.

But you don't know until you define your 'work' taxonomy, and still further, you won't know until you have a statistically significant number of runs on a given chunk of work. Once you have that, though, you can hone in on models that excel at one specific task or another and prefer those the majority of the time (say ~80%) and hold back the remainder work as a 'test corpus' just in case a different or new model does even better.

This is something I've kept in the back of my head as I've been working through my agent harness primitives -- specifically enabling different models per chunk of work.