provider fallbacks are also expensive since you lost the prompt cache. openrouter have a 'session' concept to help with this, but we ended up going direct to providers for higher rate limits and writing our own fallback logic.
So you want to use OpenRouter?
181–190 of 208 posts
Re: So you want to use OpenRouter?
#182> The same model will benchmark very differently Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware. BTW, does anyone knows if L…
> Providers probably serve quantized versions without disclosing it It should be OpenRouter's responsibility to protect you against it, by regularly benchmarking providers and giving you the control to avoid bad providers. In fact, that's a big opportunity for them, since it justifies their place as a middleman between users and inference providers.
> by regularly benchmarking providers and giving you the control to avoid bad providers.
Or by fining them, or getting rid of them altogether.
Re: So you want to use OpenRouter?
#183I was just trying deepseek v4.1 flash on OpenRouter.
After running reasonably well for a while, I return to the window and see that my scrollback is nothing but this repeated over and over again:
``` OK.
Let me write.
Let me go.
OK.
Let me write the script.
Let me go. ```
I'm not sure to what extant this is a model problem, vs some providers being fairly broken. If I chose a single provider, I could know how to blame and to avoid them. With OpenRouter, I don't know which provider I was on when this happened.
Re: So you want to use OpenRouter?
#184Yes to all this but more. The thing that made me leave and go to a single provider was token caching. I have to keep blocking providers that don't properly cache. I see performance tank and then I look in the logs and a new provider has been rotated in and every call to them is uncached because they are clearly broken. This has happened a few times now and essentially destroys cost savings (these providers also often…
We have made MASSIVE improvements here, and network-wide caching rates have been improving relentlessly. We do publish the cache rates for each endpoint; see the "performance" area of our model pages. E.g. https://openrouter.ai/deepseek/deepseek-v4-flash-0731#perfor... Open to feedback on how to make this better.
Can I block endpoints whose cache rates are below a certain percentage?
Re: So you want to use OpenRouter?
#185Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…
> We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them.
I'm glad to hear that y'all are doing this, as I was unaware that this was something OpenRouter does. I was surprised and disappointed that there are so many problematic providers that it seems like community best practice [1] is to ban somewhere in the realm of 5-6 providers. Would it be possible to provide some way to express an even stronger preference for high quality providers? E.g. "only route to first party for this model" or, "cost, but don't route to providers that more than x% worse than the first party". I'm sure something like that can be done via the API but I haven't found a UI way to do it - and having it in the UI would go a long way towards feeling like OpenRouter is looking out for me/helping solve the problem as opposed to leaving it to me to have to figure out.
[1] https://www.reddit.com/r/LocalLLaMA/comments/1mk4kt0/be_care...
Re: So you want to use OpenRouter?
#186The best part about OpenRouter is 200 OK is probably hardcoded into their responses. I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.
I was just thinking that this kind of behavior is a glimpse of the future!
Re: So you want to use OpenRouter?
#187> The same model will benchmark very differently
This is surprising to me. Does anyone have a definitive answer that accounts for this difference between providers?
Do the benchmarks that Open Router runs not account for the stochastic nature of the models? Do the providers lie about quantization or context window sizing? Does Open Router not take into account variations for a given model?
I could understand latency/cost benchmarks varying but not the actual generated token responses of the models given the exact same model parameters.
Re: So you want to use OpenRouter?
#188Earlier quoted context omitted.
I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible. For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.
Did you miss the part of this article where the benchmarks specifically call out Fireworks as one of the worst in their tests: > Fireworks scored 46% on TAU, a 30 point gap Another surprise was DigitalOcean being bottom of barrel too. Companies are apparently willing to risk their brand name by being deceptive about these heavily quantized/flawed model-serving.
It's not always deceptive. Bugs in inference can be very subtle (well, bugs everywhere can be subtle).
The providers should be benchmarking their offerings daily
Re: So you want to use OpenRouter?
#189That first diagram is striking: that deepseek's own inference is at least 5% higher on tool calling (TAU Bench) than most other providers. I wonder if they make sure their responses are valid json at the token generation level using a grammar, similar to the feature in llama.cpp.
As a regular user of openrouter I didn't know this info was available. Will have to check it out. I've definitely noticed that a high level of deepseek flash responses were looping endlessly before the 0731 release.
Edit: looks like the diagrams data is from the "Auto Exacto Benchmarks" section of the performance. Looks like they haven't run the benchmarks of deepseek flash 4.1 on the deepseek provider yet: https://openrouter.ai/deepseek/deepseek-v4.1-flash#performan...
Re: So you want to use OpenRouter?
#190Been using Olly for a while now. Very useful when you only have your Apple Watch. Didn't realize the maker had a blog.