61–70 of 70 posts
If memory serves well, I think none of them survived or had meaningful exits. I might be wrong. The main reason could be that this is a feature that gets commoditized really quickly.
So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then…
How is this different from OpenRouter? I also developed something similar for my agent harness. It's based on model predictors and an exponential moving average: https://github.com/rush86999/atom/blob/main/docs/architectur...
There is no difference. It is just another metoo copy of OpenRouter, but "Backed by Y Combinator™"
We ended up doing roughly the session-lock you're describing — pin a conversation to whichever provider is already warm, reconsider only on a cold prefix — and what surprised me is that eviction was rarely the thing that broke it.
Any request that changed the prefix at all did: a tool definition added mid-session, a system prompt carrying a timestamp, retrieval that reorders context. Those all still look like a warm session to the router and are a full re-price in practice.
So the lock ended up being less "stay until eviction" and more "stay until something upstream invalidates the prefix" — and most of our savings came from stopping our own code from invalidating it, not from the routing decision.
One interesting data point was that for smaller completions (~50 tokens or less), open-source models like OpenHands outperformed cloud-hosted Claude alternatives in terms of 95th percentile latency—likely because no network involved. But for bigger, multi-file refactoring queries, fallback to frontier models was unavoidable to hit the 90+% pass rate. The dynamic routing balanced those two extremes to keep API spend down while still scaling.
That said, the caching layer remains a bottleneck. Changes in the input context or tool states quickly cold the cache, forcing more queries to the expensive model. For workflows with lots of sequential tool calls (e.g., automated code lint + formatting + test generation), the raw switch cost was around 15-20% overhead compared to fixed-model baselines until cache warm-up happened. This matches the experiences others have reported.