Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Smart model routing directly in Claude, Codex and Cursor

github.com

Post reply on HN
Print view View original

31–40 of 127 posts

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#31

Post by adchurch »

emilio_srg2 wrote: post #28
adchurch wrote: post #17

Earlier quoted context omitted.

If you have a Claude/Codex subscription then we use that (and account for the subsidized price accordingly when making routing decisions) instead of API billing. So you get the best of both worlds: subsidized usage for frontier models + save by using open/smaller models when it's genuinely better. In practice, lots of ppl are using this to make their Claude sub limits go further!

I see but didn’t they severely limited the usage allowed with `claude -p`

But we're not routing via `claude -p`, if you have sub usage available + it's the right choice to route to a Claude model, then the router is approximately a transparent passthrough. So it gets billed like normal `claude` usage rather than `claude -p`.

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#32

Post by adchurch »

alansaber wrote: post #18

"We reward the routing model when it selects an LLM that achieves the task successfully" sounds pretty oversimplified

Indeed it is :) I skipped over talking about all the RL machinery, network design, reward function design, state representations, etc. because really the intuition is that we tell the model when it accomplishes its goal, and then it learns over time how to get better at making the right decisions in order to accomplish its goal.

Happy to talk about this in some more depth if there's anything specific you're curious about!

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#33

Post by adchurch »

k9294 wrote: post #30

What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?

Yep 100%, mentioned this in another thread (https://news.ycombinator.com/item?id=48689448) but tl;dr we build the router to be cache aware

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#34

Post by mkagenius »

We have created Murmur[1] which kind of works with your existing subscription (having API key is not mandatory). You can just tag @copilot @codex from claude code to delegate work to them. (it can also do it on its own too btw)

1. https://github.com/instavm/murmur - Murmur

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#36

Post by peterbell_nyc »

I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.

I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#37

Post by adchurch »

mkagenius wrote: post #34

We have created Murmur[1] which kind of works with your existing subscription (having API key is not mandatory). You can just tag @copilot @codex from claude code to delegate work to them. (it can also do it on its own too btw) 1. https://github.com/instavm/murmur - Murmur

Very interesting - curious how you've used it yourself so far? I can imagine one use case would be having e.g. GPT 5.5 review Opus 4.8's work?

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#39

Post by adchurch »

peterbell_nyc wrote: post #36

I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session. I'm just trying to figure out why on the fly…

[dead]

Re: Show HN: Smart model routing directly in Claude, Codex and Cursor

#40

Post by echelon »

alansaber wrote: post #19
adchurch wrote: post #7

Earlier quoted context omitted.

You're right and that's why we built the router to be cache aware! Once it starts using one model, the threshold to switch to another model will be higher because the additional cost of the cache miss needs to be worth the cost savings or quality increase. This is the key thing that other routers we've seen miss: they're stateless so for a coding agent use case you end up spending more money due to all the cache miss…

That is interesting, sounds like in practice you only end up routing between 2 models

Or not routing at all.

In practice you just pick one and stick with it until the API stops or you hit performance issues.

Post reply on HN