61–69 of 69 posts
If memory serves well, I think none of them survived or had meaningful exits. I might be wrong. The main reason could be that this is a feature that gets commoditized really quickly.
So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then…
How is this different from OpenRouter? I also developed something similar for my agent harness. It's based on model predictors and an exponential moving average: https://github.com/rush86999/atom/blob/main/docs/architectur...
There is no difference. It is just another metoo copy of OpenRouter, but "Backed by Y Combinator™"
We ended up doing roughly the session-lock you're describing — pin a conversation to whichever provider is already warm, reconsider only on a cold prefix — and what surprised me is that eviction was rarely the thing that broke it.
Any request that changed the prefix at all did: a tool definition added mid-session, a system prompt carrying a timestamp, retrieval that reorders context. Those all still look like a warm session to the router and are a full re-price in practice.
So the lock ended up being less "stay until eviction" and more "stay until something upstream invalidates the prefix" — and most of our savings came from stopping our own code from invalidating it, not from the routing decision.