Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Smart model routing directly in Claude, Codex and Cursor

github.com

Post reply on HN
Print view View original

71–80 of 127 posts

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#71

Post by adchurch »

nikcub wrote: post #62

I'm glad there are more attempts at solving model routing, as costs (at API rates) has really become an issue. Some feedback: 1. Reiterate the cache issue from other comments already here. there is a lot of optimisation in harnesses around caching and a proxy model blows that up 2. Coding agents are model aware - they already route code discovery to mini / flash models, planning to heavy models, workflow design to ul…

Really appreciate the thoughtful feedback!

1. Agree it's important, fwiw the proxy model doesn't blow this up though - only incurs a 1 time cost when switching models and we're aware of that when making routing decisions

2. The agents are model aware yes but they are not incentivized to optimize too heavily here (in particular they don't use OS models even when they would be better). I think that's where this router comes in and brings genuine improvement.

3. Two parts here: 1 is continuing to grow our golden dataset over time, 2 is using reward signals from production traffic (on a per-customer basis or, if allowed, across all users)

4. Yes we have these internally, great callout that we should publish! Will do + will link from the repo soon. (Fwiw I think these benchmarks are useful but don't fully capture vibes - you should try it out yourself for that!)

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#72

Post by gopher_space »

GodelNumbering wrote: post #58
adchurch wrote: post #56

Earlier quoted context omitted.

When we started building this we did it as an experiment and we thought the same thing might be true (cache misses would make the whole thing pointless). This turned out not to be true! I think there are 3 reasons intuitively: 1. Small models can carry out a good number of requests e2e 2. Small model for part of a request + cache miss For our own usage we've saved 40% so far (that is of course including costs of unca…

This assumes a perfect problem routing though. Determining the complexity class of an arbitrary problem is generally undecidable or extremely hard (Rice's theorem implication). So, in real use cases, you need to amortize all cases where the problem got routed to the wrong model and recovery had to be performed) For example, if my task was "refactor this component to decouple all messy nesting", the problem router can…

In my mind one of the problems is that I'm using the term 'router' to describe something more akin to a train schedule. A list of abilities, cost, and timeframe to be used by a model capable of deconstructing its own process.

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#73

Post by matt_d »

Looks interesting!

Out of curiosity, how does it compare with vLLM Semantic Router?

For reference:

https://vllm-semantic-router.com/

https://github.com/vllm-project/semantic-router

vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models, https://arxiv.org/abs/2603.04444

https://github.com/vllm-project/semantic-router

For instance, does it offer similar algorithms:

- vllm-sr/auto: efficient, fast, balanced routing, similar in spirit to Fugu // Sakana Fugu — Multi-Agent System as a Model: https://sakana.ai/fugu/ - vllm-sr/fusion: panel-style multi-model reasoning and synthesis. - vllm-sr/flow: router-native workflow orchestration - vllm-sr/remom: multi-round reasoning over one or multiple models.

FWIW, it does look good on https://routeworks.github.io/leaderboard

Ref.

RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers, https://arxiv.org/abs/2510.00202, https://github.com/RouteWorks/RouterArena

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#75

Post by gopher_space »

peterbell_nyc wrote: post #36

I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session. I'm just trying to figure out why on the fly…

"Based on your subscription tier and local hardware here's a list of models that fit and process definitions your biggest brain will comfortably handle."

I guess that sounds a lot like moving your evals and auto tunes to a third-party, but I don't have the time, budget, or inclination to create a system like this out of whole cloth and then keep it relevant.

I could see something that provides on-the-fly routing information being useful, but actual decision-making is too dependent on context.

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#76

Post by adchurch »

notatoad wrote: post #74

Is this talking to claude code, or to claude api (and paying api rates)? programatically routing requests through claude code sounds like a good way to get banned, just like the opencode and openclaw users.

If you have a Claude sub with subsidized usage we use that. If not you pay API prices.

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#77

Post by gopher_space »

treexs wrote: post #59

Ahh been working on the same thing for a while now but haven't launched yet

A lot of people are working on the same thing because nobody's come up with a definition of "thing" that people agree on yet. Your project would be valuable just for adding another point of view to the conversaion.

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#78

Post by adchurch »

lubujackson wrote: post #64

I notice Cursor already does something similar. Even if I have Opus 4.8 selected, it will trigger subagents using Composer 2.5. I like using Auto personally because it is pretty effective and deeply discounted, but at work I YOLO Opus high. I imagine a solution like this will eventually be an enterprise-forced solution because there is no reason right now for individual developers to be selective about model pricing.…

Oh interesting, didn't know Cursor did that! Totally makes sense though, routing subagents is def the easiest win, no need to have any cache awareness.

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#79

Post by adchurch »

jmalicki wrote: post #60

> with no noticeable differences in quality or velocity. Have you done any A/B tests on this with evidence? (That's one thing I'd be very interested to see for claims like this - I'm not necessarily doubting you, it just seems like it could be useful to understand claims of quality/efficiency)

Great question! Our main product quantifies engineering productivity & quality so I think we're uniquely qualified to answer this - our velocity has only gone up and our quality (bugs introduced, code turnover) has not budged per our own analysis.
Post reply on HN