Now that even the smaller models from labs (Luna, deepseek v4 flash) are getting powerful, I think the orchestrator pattern of a smart model coordinating smaller models for work will end up being the way to go.
Everyone is building LLM routers, we deprecated ours
41–50 of 94 posts
Re: Everyone is building LLM routers, we deprecated ours
#42Insider take: routing will not be a (successful, durable) thing, at least not externally to model providers. The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut i…
This arbitrage isn't durable either though, as it's just a race to the bottom with open research, and the long term destination of everything is custom models for different tasks, since it's going to become increasingly apparent that some areas of knowledge have anti-synergy as we push up the diminishing return curve.
Re: Everyone is building LLM routers, we deprecated ours
#43What was the architecture of your router? If it was based on GRPO/RL, it would be interesting to hear why your router performance capped.
I think the truth is that it's not an efficient cost cutting method. Your router has to be at least as 'smart' as all the but the smartest of your models (models do poorly when asked 'is this a task you're well suited to'), and that means you're caching multiple prompt histories including kv-filling/prefix caching on your expensive router model. Most of the time, not super great for savings.
Re: Everyone is building LLM routers, we deprecated ours
#44> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…
You don't just try every model that comes along... and for the ones you do, you have eval pipelines set up. It's engineering, you have to test stuff works.
Now, there are thousands of use cases for lots of the fontier models. I don't use them for everything, so I don't test them except on the stuff I'm using them for.
Re: Everyone is building LLM routers, we deprecated ours
#45To add: I doubt frontier labs will build routers - they are not financially incentivized to optimize token usage (though in the short term they may be incentivized by constrained GPU capacity to reduce load).
Re: Everyone is building LLM routers, we deprecated ours
#46> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…
Just don’t switch models every week and you’ll be fine. Especially in production.
Re: Everyone is building LLM routers, we deprecated ours
#47nobody else is putting a router inside a Trusted Execution Environment https://trustedrouter.com/
Re: Everyone is building LLM routers, we deprecated ours
#48I think the author need to rethink out of the box what's LLM routers in a traditional sense ( input in, route, output ) and move to think how a router would work in agentic workflow. See cognition Devin Fusion design.
Re: Everyone is building LLM routers, we deprecated ours
#49[deleted]
Re: Everyone is building LLM routers, we deprecated ours
#50The 'personality' of the model matters a lot even for stuff like Code Gen. If you're building something with Fable and then get routed to Opus cause the router decided your task is not 'demanding enough' it's like: great, now I have to read complex prose like "That's the thing, and the thing is the point:..." Opus-isms when I just wanted to make a shopping cart