Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to use OpenRouter?

mmoustafa.com

Post reply on HN
Print view View original

211–220 of 222 posts

Re: So you want to use OpenRouter?

#211

Post by bbor »

lelandbatey wrote: post #145
bbor wrote: post #41

Earlier quoted context omitted.

if you want to control the routing, you'd lock down providers I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled…

> There's not even a way to compare providers, AFAICT That's not quite true. The only thing they don't show per-provider is benchmark data, cause I don't think they are doing continuous benchmarking of each model from each provider, as I assume they feel that's too expensive. You can see hugely detailed breakdowns for near-time metrics per provider for any model by visiting the page for that model on Openrouter. For…

No I know, I was in the GUI as I wrote that lol. As the other person said: if they vary this much in quality, not including that way above updtime and performance is absurd. What would you use a fast, always-up, broken endpoint for?

Re: So you want to use OpenRouter?

#212

Post by bbor »

numlocked wrote: post #87
bbor wrote: post #36

Wait WTF?! I thought performance was, well, performance, not efficacy ! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I have no idea where the author got those clean charts. "AutoExacto" numbers (completely meaningless name) are available for 6 providers, two of which are the same provider, and none of which are even in the top half throughput-wise. When you click "+28 more pr…

We run benchmarks against all of our endpoints, in production. That first chart that the author shows is in fact our live benchmarking data. If providers underperform, we kick them out of the routing pool. That is why we run those benchmarks. Performance And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of va…

Oh wow someone from the team! I'm really heartened by this response -- as you can tell, I've been something of a fan, so I felt kinda betrayed by the idea that this was a risk. Thanks for clarifying, and for taking my frustrated hyperbole in stride.

Submitting an application to your product role now. "All of the models are at least okay" is heartening, but there's a whole bunch of fun places to take this.

¡Viva La OpenRouter! (again)

Re: So you want to use OpenRouter?

#214

Post by ndr_ »

binarymax wrote: post #104

Pretty damning. We're using OpenRouter for some research tasks and it's making me question everything. I suspect lots of the model providers are running into issues like these: https://forum.level1techs.com/t/why-your-local-llm-feels-dum... (HN discussion: https://news.ycombinator.com/item?id=49402232 )

If you require consistency for research, I wouldn't recommend relying on hosted inference without validating the provider/inference stack very carefully.

For my contribution to OpenAI's gpt-oss red-teaming competition (published as arXiv:2510.01259), I initially used OpenRouter with DeepInfra and Together AI. The results were much too noisy to draw reliable conclusions from, and markedly different behavior through AWS Bedrock was the last straw. I ended up renting GPUs through vast.ai and running the model myself with vLLM.

Re: So you want to use OpenRouter?

#215

Post by ndr_ »

epistasis wrote: post #101
numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

Thanks for the insight here. One thing to note about the first graph: nobody is doing as well as the first part on tool calling, and it's not close. This might be the fault of the other providers, but it's probably just something slightly different that the first party does with the model inference program than anybody else, and that's not sure to weights it's due to vLLM twiddling (or whatever) and probably becuase…

This matches what I found experimentally with gpt-oss-20b during OpenAI's red-teaming challenge. After moving from hosted inference to running the model myself on rented H100s via vast.ai, I saw the model refuse the same kinds of prompts at noticeably different rates depending on the inference stack — differences of roughly 5–10 percentage points with otherwise identical experimental parameters and seeds.

So I very much agree that this isn't necessarily about providers secretly changing the weights. For reproducible work, the serving stack - engine, version, hardware, configuration, and probably more - really belongs in the methodology alongside the model itself.

I wrote up the results here: “In AI Sweet Harmony” (arXiv:2510.01259).

Re: So you want to use OpenRouter?

#216

Post by creativeSlumber »

> 6. 200 OK, no answer Reasoning models sometimes put everything in the reasoning field and hand back content: null, finish_reason: "stop". 345 completion tokens, HTTP 200, nothing to show the user. A 200 tells you the request was served, not that there's an answer in it. No content and no tool call is a failure, throw and retry.

I ran into this so much I had to modify my harness to handle this.

Re: So you want to use OpenRouter?

#217

Post by kroaton »

numlocked wrote: post #169
jmward01 wrote: post #105

Yes to all this but more. The thing that made me leave and go to a single provider was token caching. I have to keep blocking providers that don't properly cache. I see performance tank and then I look in the logs and a new provider has been rotated in and every call to them is uncached because they are clearly broken. This has happened a few times now and essentially destroys cost savings (these providers also often…

We have made MASSIVE improvements here, and network-wide caching rates have been improving relentlessly. We do publish the cache rates for each endpoint; see the "performance" area of our model pages. E.g. https://openrouter.ai/deepseek/deepseek-v4-flash-0731#perfor... Open to feedback on how to make this better.

You need to validate what providers are actually serving. Add benchmarks, properly showcase what quantization they are serving on the model and KV cache, etc.

Until that happens, your service is doomed to be shitty.

Re: So you want to use OpenRouter?

#218

Post by bellowsgulch »

numlocked wrote: post #170
radicality wrote: post #124

Earlier quoted context omitted.

Oh, interesting, I feel like a year so ago I read something posted from openrouter team that they write that credits expire, but that they in actuality don’t expire them. But maybe I’m misremembering, or perhaps that’s changed for the worse in last few months :S

We started actually expiring the credits a ~month ago. If you make any kind of API request, it resets the clock. We try to make it a very generous policy, but we can't keep a monotonically increasing liability on the books. We end up owing (a lot) of taxes on it, but can't actually recognize revenue. We would much rather you spend the credits! Hence the reminder emails, and generous "clock reset" policy.

Absolutely no where is that shown to customers, and OpenRouter is going to be in a world of shit when people learn this by HN post and the business asks themselves why its reputation keeps taking hits.

What a fucking ridiculous thing for me to learn by social news.

Re: So you want to use OpenRouter?

#219

Post by epistasis »

ndr_ wrote: post #215
epistasis wrote: post #101

Earlier quoted context omitted.

Thanks for the insight here. One thing to note about the first graph: nobody is doing as well as the first part on tool calling, and it's not close. This might be the fault of the other providers, but it's probably just something slightly different that the first party does with the model inference program than anybody else, and that's not sure to weights it's due to vLLM twiddling (or whatever) and probably becuase…

This matches what I found experimentally with gpt-oss-20b during OpenAI's red-teaming challenge. After moving from hosted inference to running the model myself on rented H100s via vast.ai, I saw the model refuse the same kinds of prompts at noticeably different rates depending on the inference stack — differences of roughly 5–10 percentage points with otherwise identical experimental parameters and seeds. So I very m…

Great concrete experience!

I'd love to see some published token log_probs for given prompts and seeds that accompany a model card. Not sure if that's enough, thigh.

But then it's been so long since I dealt with the api directly that I don't even know if modern models show the top tokens and probabilities any more.

Re: So you want to use OpenRouter?

#220

Post by chr15m »

numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

What a refreshing response. I'm a big fan and recommend your service frequently to people locked in to providers. Godspeed!
Post reply on HN