[dead]
So you want to use OpenRouter?
141–150 of 208 posts
Re: So you want to use OpenRouter?
#142Earlier quoted context omitted.
> Providers probably serve quantized versions without disclosing it It should be OpenRouter's responsibility to protect you against it, by regularly benchmarking providers and giving you the control to avoid bad providers. In fact, that's a big opportunity for them, since it justifies their place as a middleman between users and inference providers.
100% agree. The key issue is that users think they are getting results from a specific model configuration and they are clearly not, which is fraudulent. If OpenRouter wants to succeed as a business, they need to be auditing the providers they connect to (i.e. benchmarking) and removing fraudulent ones from their service.
They already succeed selling for 7 billion. Auditing is up to you and can be a new business for those looking for new opportunties.
Re: So you want to use OpenRouter?
#143> Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user.
This isn't necessarily an OpenRouter issue, Google's Gemini will sometimes do it on their own API. Since they summarize reasoning this means you pay the whole cost and can't get anything out of it.
Re: So you want to use OpenRouter?
#144Earlier quoted context omitted.
> I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. Why do you care about the public benchmarks at all? The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use…
If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity. I don't think anyone serious is just willy-nilly making individual requests against OpenRouter Despite your confidence, that is indeed the basis of this massive corporations entire business plan. If you just want be…
I think you're focusing only on the general coding agent aspect of LLMs.
> If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity.
That is the way LLMs have to be used for highest reliability. While the term stochastic parrot has been co-opted by unreasonable LLM skeptics, that is indeed what LLMs are. You have to have grounded evals that check outcomes if you want to use them reliably - or a human in the loop works too.
The more general your family of tasks, the less likely you can make automated evals. So for "general" coding agents, you need a human in the loop that can verify and it's not that easy to write an eval.
But if you have specific tasks, then you can spend the time to make a eval, and then you can optimise the way you use the LLM and get extremely good success rates. It's not like it's black magic. Nor does it need large budgets.
> that is indeed the basis of this massive corporations entire business plan.
No. Individual developers using codex (for extremely underspecified general engineering) needs human in the loop, is not amenable to evals but is only a fraction of all LLM usecases.
Re: So you want to use OpenRouter?
#145Earlier quoted context omitted.
> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, i…
if you want to control the routing, you'd lock down providers I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled…
> There's not even a way to compare providers, AFAICT
That's not quite true. The only thing they don't show per-provider is benchmark data, cause I don't think they are doing continuous benchmarking of each model from each provider, as I assume they feel that's too expensive. You can see hugely detailed breakdowns for near-time metrics per provider for any model by visiting the page for that model on Openrouter. For example see the page for Qwen 3.8 27B: https://openrouter.ai/qwen/qwen3.8-27b
Some of the killer stats they show per provider:
- Pricing: Effective price accounting for cache hit rate, by provider
- Performance: Throughput in tok/s, latency, E2E latency, tool call error rate, structured output error rate, and more; all per provider.
- Uptime: You have to click on the provider to see their specific uptime, but doing so does show the last-7-days uptime, and you can click to see more.
Re: So you want to use OpenRouter?
#146I still don't get the appeal of OpenRouter. Why not just generate API keys from the providers you want to use, which should not exceed 3-4, I assume, and integrate them into your apps to call them? Are people so lazy, or am I missing something?
Re: So you want to use OpenRouter?
#147Great post. I had realized there is variation, but the charts are much worse than expected
Re: So you want to use OpenRouter?
#148I still don't get the appeal of OpenRouter. Why not just generate API keys from the providers you want to use, which should not exceed 3-4, I assume, and integrate them into your apps to call them? Are people so lazy, or am I missing something?
If you're making a commercial SaaS, that's probably the way to go. For individual users like me, with a coding harness and some extra BYOK tools, OpenRouter is convenient and the few extra percent don't hurt much. I appreciate being able to try out any new model with just an ID swap, same with inference providers when they prices change. I know I wouldn't enjoy managing 5+ accounts, each with their own balance, in 3+ tools, so this is one thing I'm happy to outsource.
Re: So you want to use OpenRouter?
#149[deleted]
Re: So you want to use OpenRouter?
#150Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…
I have attempted to report numerous errors in your Chat UI, but you seem to have decided that customer input is undesired, as you have an AI support system that ignores my problem and refuses to escalate to anyone or create a ticket.
To this day, your in-chat “Report an Issue” button still does not work consistently, and I am still billed for empty responses from many image providers.