Everyone is building LLM routers, we deprecated ours

Post by overgard » Sat, Aug 01, 2026, 8:43 PM UTC

danielmarkbruce wrote: post #90

overgard wrote: post #61

Earlier quoted context omitted.

Well, first off, I don't need to test that stuff works, the labs need to do that! I can afford to wait a couple days to get an impression on how people like a model before I switch. I don't want to be a beta tester. But honestly, even if I went with your idea of picking a small number of models to eval, I wouldn't even know how to choose that subset other than.. what's hottest in the hype cycle? The benchmarks are us…

You are completely wrong and will never make a good product/service with that attitude. You need to test the models work for your use case. Knowing how much of type X data a model was trained on isn't going to help you one bit. Various phases of training can effectively wipe out the training from earlier steps if not done well. All that matters is does the model do what you want it to do. In practice almost everythin…

Post by lucas_t_a » Sun, Aug 02, 2026, 12:02 AM UTC

all2 wrote: post #40

dweez wrote: post #18

I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori. One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" ( https://en.wikipedia.org/wiki/Busy_beaver…

I think routing should be pushed 'down the stack' so to speak. What I mean is that most tasks can be recursively fragmented into smaller tasks, and once you've hit suitable leaf nodes -- where the task is very granular -- you can begin to deterministically show which models perform better or worse for that specific task. Then, when your agent is running a workflow, you may use various models for different steps in a…