71–80 of 94 posts
>Our router was classifying each request into one of four different tiers of complexity: simple, standard, complex and reasoning. Seems like a naive classification model lacking context?
I'm still working on mine: https://github.com/rush86999/atom/blob/main/docs/architectur...
To add: I doubt frontier labs will build routers - they are not financially incentivized to optimize token usage (though in the short term they may be incentivized by constrained GPU capacity to reduce load).
Insider take: routing will not be a (successful, durable) thing, at least not externally to model providers. The labs are incentivized to solve this problem themselves, since they’re competing on a 2D cost-intelligence frontier. If they can reduce cost without harming intelligence they will do that and pass on (some of) the cost reduction to the user. There are nicer solutions available to them because they can cut i…
What was the architecture of your router? If it was based on GRPO/RL, it would be interesting to hear why your router performance capped. I think the truth is that it's not an efficient cost cutting method. Your router has to be at least as 'smart' as all the but the smartest of your models (models do poorly when asked 'is this a task you're well suited to'), and that means you're caching multiple prompt histories in…
The 'personality' of the model matters a lot even for stuff like Code Gen. If you're building something with Fable and then get routed to Opus cause the router decided your task is not 'demanding enough' it's like: great, now I have to read complex prose like "That's the thing, and the thing is the point:..." Opus-isms when I just wanted to make a shopping cart
> Complexity cannot be deduced from the prompt alone. Let’s take an example: “evaluate the tests for the repo $GIT_REPO and improve them” can be a very simple task if you mention a personal website written in plain HTML5; or an incredibly complex task if you target the Linux kernel repo. I don't think this a good example. The first step would be reading the documentation, reading an overview of the tests, then execut…
I agree with one distinction - coding agent workflows can use defined subagent roles that are pinned to specific models and I have found this very effective. The orchestrator is building all the context to make these assignments - it’s not a dumb router. Using Minimax M3 for exploration and librarian tasks for example is fast and cheap - my $10 plan lasts all month and saves a lot of tokens for my main coding plan.
Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence: > "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models. I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially…