Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to use OpenRouter?

mmoustafa.com

Post reply on HN
Print view View original

171–180 of 208 posts

Re: So you want to use OpenRouter?

#171

Post by numlocked »

lelandbatey wrote: post #145
bbor wrote: post #41

Earlier quoted context omitted.

if you want to control the routing, you'd lock down providers I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled…

> There's not even a way to compare providers, AFAICT That's not quite true. The only thing they don't show per-provider is benchmark data, cause I don't think they are doing continuous benchmarking of each model from each provider, as I assume they feel that's too expensive. You can see hugely detailed breakdowns for near-time metrics per provider for any model by visiting the page for that model on Openrouter. For…

We are doing continuous benchmarking of each endpoint, for each provider, and it is very expensive :)

Re: So you want to use OpenRouter?

#172

Post by johnsmith1840 »

mmoustafa wrote: post #168
johnsmith1840 wrote: post #159

Same experience. I migrated over and am happy but that intro was brutal. My basics are I have a test suite that: 1. Finds newest models of my versions 2. Inferences every single model and a few providers for each with a short problem 3. Analyze latency and if a model failed the stupid simple questions drop it and the provider 4. Run larger context haystack kinds of problems. It's cheap and fast less than 5$ so I can…

Yes, you hit it on the head. It's worth it, but you have to do your own evaluation and promotion. Example I forgot to mention: `:nitro` ranking is not the fastest, I do a round robin sampling with representative payloads to find out the fastest providers and reorder my list on the fly.

Your doing that sampling/promotion automatically?

I considered an automatic promotion path but decided against it I want to actually review the data myself first.

What's your model churn rate like? I was worried about customer experience by same day maybe same work getting a totally different model response (also caching is worse)

Re: So you want to use OpenRouter?

#173

Post by mmoustafa »

numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

Thanks for the response Chris, I couldn't make my business work without OpenRouter in the first place, so kudos

This was meant as more of a technical reference, sorry you had to wake up to a PR drill lol

Re: So you want to use OpenRouter?

#174

Post by numlocked »

noahbp wrote: post #150
numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

I have attempted to report numerous errors in your Chat UI, but you seem to have decided that customer input is undesired, as you have an AI support system that ignores my problem and refuses to escalate to anyone or create a ticket. To this day, your in-chat “Report an Issue” button still does not work consistently, and I am still billed for empty responses from many image providers.

Pulled some data - a surprisingly large number of requests to models that generate images, where the user seems to want an image, do NOT return an image. In our chatroom it's ~11%! However, in almost all of those cases, the model is instead deciding to return text, and you are being billed (correctly) for that text. We are not billing you for an image that wasn't returned.

Concretely, we pulled data on the last few days of image gen requests in our chatroom (50,893 requests). 5,846 got a text response (which is frustrating, I'm sure) and were billed for text appropriately. The model did not generate an image.

There were 16 requests where a customer was billed, but neither an image or text was returned. Those should not have been charged, and we'll see if we can either fix that issue or ensure that customers aren't charged.

Re: So you want to use OpenRouter?

#175

Post by numlocked »

mmoustafa wrote: post #173
numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

Thanks for the response Chris, I couldn't make my business work without OpenRouter in the first place, so kudos This was meant as more of a technical reference, sorry you had to wake up to a PR drill lol

No stress! My first reaction to this was OMG THIS IS AN INCREDIBLE RESOURCE!! We are 100% grateful for this sort of feedback! Here is a direct quote of what I said at 8:17am this morning when someone sent me the article and I scanned it:

  this is amazing!!
  [8:19 AM]The first obvious win is routing around providers that arent handling image inputs correctly. That should be straightforward
  [8:19 AM]The effort param stuff...I thought we had addressed that, but will dig in. This is incredible feedback
  [8:20 AM]We should hire this guy.
Our goal is to get better, fast!

Re: So you want to use OpenRouter?

#176

Post by kelvinjps10 »

embedding-shape wrote: post #34
joshstrange wrote: post #20

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “Ope…

> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, i…

I mean that should be their job to verify the quality of the providers they have in their platform. Then what is the 5% for?

Re: So you want to use OpenRouter?

#177

Post by dnugget »

_ink_ wrote: post #14

Wow, that explains a lot. I was using OpenClaw with open router, but stopped because how inconsistent the results were over the same prompt. I thought OC is at fault, never would have guessed that there are differences between providers.

[dead]

Re: So you want to use OpenRouter?

#178

Post by wren6991 »

numlocked wrote: post #170
radicality wrote: post #124

Earlier quoted context omitted.

Oh, interesting, I feel like a year so ago I read something posted from openrouter team that they write that credits expire, but that they in actuality don’t expire them. But maybe I’m misremembering, or perhaps that’s changed for the worse in last few months :S

We started actually expiring the credits a ~month ago. If you make any kind of API request, it resets the clock. We try to make it a very generous policy, but we can't keep a monotonically increasing liability on the books. We end up owing (a lot) of taxes on it, but can't actually recognize revenue. We would much rather you spend the credits! Hence the reminder emails, and generous "clock reset" policy.

> but can't actually recognize revenue

Hmm? You have the revenue already. I know it's awkward from an accounting point of view, but you already took my money. "Letting" me keep the balance in the account is not generous.

Edit: on re-reading this came out more combative than I intended, sorry. I think what you're doing is reasonable.

Re: So you want to use OpenRouter?

#179

Post by wren6991 »

Yeah, this is pretty accurate. Some providers are basically scams too. I encountered one provider for GLM-5.2 which ran at 200 tps (absurdly high), and was so broken that it would issue 20 full reads of the same file in one turn and quickly jump up to 300~400k context and charge me for the prefill. Went straight on my deny-list, but they got a few dollars out of me first.

Another common annoyance is having a request go to a provider that dribbles out ~1 tps (even for small models like DeepSeek V4 Flash). If you cancel the request, you still get charged for the prefill and the handful of generated tokens. If you don't cancel the request, you might be waiting 10 minutes for the turn to finish.

The overall experience is pretty good, and it's the best way to try new models, but they don't appear to do any real vetting or apply any quality standards to their providers, and occasionally it bites you.

Re: So you want to use OpenRouter?

#180

Post by pessimizer »

embedding-shape wrote: post #68
Spacemolte wrote: post #65

Earlier quoted context omitted.

1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them. 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite? 3. Great example, if amazon does not lead…

> 1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them. OpenRouter does reliably route to your specified model and provider, otherwise it'd pretty much be fully broken. Parent is complaining about the auto-provider chosing, not that all providers are unreliable.…

> Parent is complaining about the auto-provider chosing, not that all providers are unreliable.

Parent is claiming that the choosing is unreliable. If I rely on a provider to bring me tuna to some spec, but they get it from many different fishermen, it doesn't mean that the tuna doesn't have meet the spec. The complaint is that they're given a bunch of knobs that simply don't work with providers that they could be switched to. That's like saying that I want my tunas to be 20lbs. minimum, and I get switched to a provider that doesn't weigh their tuna at all.

The choosing is all OpenRouter provides. If it doesn't do that, then what is it good for? If I have to permanently pin the one provider who doesn't ignore what I've asked, why shouldn't I just deal with them directly?

edit: it's really supposed to reduce providers to a commodity market. If you're selling e.g. produce to a commodity market, you can't just ship whatever the hell you want. You ship something indistinguishable from others, or more likely the market itself allows you to grade what you're shipping so it's put into a bin with virtually identical stuff. The customer just buys Grade B Wheat.

Post reply on HN