Viewing profile — adchurch

Effectively yes (based on cost though, not raw token count)

We trained a model to select which LLM to call at any given turn, based on lots of agent traces

Yes the open source models are very good, that’s a big part of what makes this router save so much money in practice! There definitely are some things they still don’t handle well …

Yes we can route to Gemini models too and we handle all the translation complexity there!

We welcome the competition :)

Yep exactly

Yes because it's a model explicitly trained to make model selections! Opus probably doesn't have a great idea of when to send a task to DeepSeek vs. to Sonnet, for example.

We haven't experimented with routing to local LLMs much. Technically they benefit from the cache too although it's more a question of latency than cost. But tbh I haven't seen grea…

I think the key detail here is that we use embeddings of the prompt + previous context in order to decide where to route the request, and if one model is getting stuck we can cours…

We consider the cost of missing the cache when making each routing decision after the initial one. Discussed in a bit more depth here: https://news.ycombinator.com/item?id=48689448…

Good questions. From what I can tell, vLLM semantic router is more optimized for one-off prompt/response workflows rather than agentic coding (I don't think it's cache aware). As a…

Cool, interested to see your approach when you do launch! I think it's a really interesting problem

Great question! Our main product quantifies engineering productivity & quality so I think we're uniquely qualified to answer this - our velocity has only gone up and our quality (b…

Oh interesting, didn't know Cursor did that! Totally makes sense though, routing subagents is def the easiest win, no need to have any cache awareness.

If you have a Claude sub with subsidized usage we use that. If not you pay API prices.

Really appreciate the thoughtful feedback! 1. Agree it's important, fwiw the proxy model doesn't blow this up though - only incurs a 1 time cost when switching models and we're awa…

Appreciate the kind words! Lmk if you have any feedback on it from using!

I would argue they do not have a good incentive to build this and make it better. Why would Anthropic route Claude Code traffic to DeepSeek (at 20% of the cost)?

Very important consideration, addressed it in another thread ( https://news.ycombinator.com/item?id=48689448 ). tl;dr we built this to be cache aware for exactly this reason

When we started building this we did it as an experiment and we thought the same thing might be true (cache misses would make the whole thing pointless). This turned out not to be …

Fair enough, not meant to be marketing just a statement of fact. Would have turned me off too 18 months ago but times change...

It's a real concern! We take this stuff super seriously ( https://trust.mycroft.io/weave ) and tbh most of our customers opt for the hosted version because it's much simpler on the…

Yep cache awareness is super important, mentioned this in another thread here: ( https://news.ycombinator.com/item?id=48689448 ) But intuitively I think it makes sense that a model…

I guess delivering business value is always #1, I just meant it's the biggest problem they're trying to solve. Here's a recent example that was public: https://fortune.com/2026/05/…

The choice on the first turn is super important for this reason! But if a user prompt sends the convo in a very different direction then often it does make sense to reroute at that…