Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Smart model routing directly in Claude, Codex and Cursor

github.com

Post reply on HN
Print view View original

81–90 of 127 posts

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#81

Post by adchurch »

matt_d wrote: post #73

Looks interesting! Out of curiosity, how does it compare with vLLM Semantic Router? For reference: https://vllm-semantic-router.com/ https://github.com/vllm-project/semantic-router vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models, https://arxiv.org/abs/2603.04444 https://github.com/vllm-project/semantic-router For instance, does it offer similar algorithms: - vllm-sr/auto: efficient…

Good questions. From what I can tell, vLLM semantic router is more optimized for one-off prompt/response workflows rather than agentic coding (I don't think it's cache aware).

As another commenter (https://news.ycombinator.com/item?id=48689994) pointed out, for one-off requests, I think it makes more sense to lock to one model whose behavior you understand very well. For dynamic requests like the ones going to a coding agent I think dynamic routing makes more sense but it does need to be cache aware.

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#82

Post by adchurch »

pradeep1177 wrote: post #70

So, how are you handling read/write caching? I mean, if I keep routing the next prompt based on the task weights? How about if I'm sending every 5th query to opus, which do expensive write cache?

We consider the cost of missing the cache when making each routing decision after the initial one. Discussed in a bit more depth here: https://news.ycombinator.com/item?id=48689448

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#83

Post by adchurch »

GodelNumbering wrote: post #58
adchurch wrote: post #56

Earlier quoted context omitted.

When we started building this we did it as an experiment and we thought the same thing might be true (cache misses would make the whole thing pointless). This turned out not to be true! I think there are 3 reasons intuitively: 1. Small models can carry out a good number of requests e2e 2. Small model for part of a request + cache miss For our own usage we've saved 40% so far (that is of course including costs of unca…

This assumes a perfect problem routing though. Determining the complexity class of an arbitrary problem is generally undecidable or extremely hard (Rice's theorem implication). So, in real use cases, you need to amortize all cases where the problem got routed to the wrong model and recovery had to be performed) For example, if my task was "refactor this component to decouple all messy nesting", the problem router can…

I think the key detail here is that we use embeddings of the prompt + previous context in order to decide where to route the request, and if one model is getting stuck we can course correct and move to a different model.

So: we can reasonably cluster similar problems together and learn how models handle them, and the entire system doesn't fail if the initial decision is off.

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#85

Post by adchurch »

thandv wrote: post #84

This might be a stupid question, but can a extra added local llm help with the caching problem?

We haven't experimented with routing to local LLMs much. Technically they benefit from the cache too although it's more a question of latency than cost. But tbh I haven't seen great results in the wild from working with local LLMs for coding - curious if you've had any success with them?

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#87

Post by nativeit »

This would have been neat back when I could afford enough tokens to even set it up properly. Now I’ve had to increase my GH Copilot subscription just to cover the bare minimum updates to a few websites every month, and I no longer do any test driving, or even recreational coding projects. I don’t have hundreds of dollars a month to plow into these products, so I’m rationing use, looking for better local options, and being much more discerning about where these tools actually save time. Precarious time to be alive…

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#88

Post by newaccountman2 »

nativeit wrote: post #87

This would have been neat back when I could afford enough tokens to even set it up properly. Now I’ve had to increase my GH Copilot subscription just to cover the bare minimum updates to a few websites every month, and I no longer do any test driving, or even recreational coding projects. I don’t have hundreds of dollars a month to plow into these products, so I’m rationing use, looking for better local options, and…

try OpenCode

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#89

Post by asdev »

adchurch wrote: post #67
asdev wrote: post #66

Large model companies will likely build this and make it better. It'll also be cheaper overall since they'll be subsidizing token cost if you use them directly vs third party router paying API costs

I would argue they do not have a good incentive to build this and make it better. Why would Anthropic route Claude Code traffic to DeepSeek (at 20% of the cost)?

They'll route traffic to Haiku or one of their cheaper models, not third parties. Overall cost will end up being cheaper than whatever you are doing
Post reply on HN