Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to use OpenRouter?

mmoustafa.com

Post reply on HN
Print view View original

111–120 of 208 posts

Re: So you want to use OpenRouter?

#111

Post by fzysingularity »

The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id).

If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestration in providers like OR, that leave the user more confused about the model’s capabilities.

Re: So you want to use OpenRouter?

#112

Post by pingtoven »

numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

toven from openrouter, leading the team working with our providers - very interested in some of the things found in the report, I dug in to the image failures specifically, and in that case we have data showing deepinfra was correctly parsing images when the endpoint went live in july, but today fails those tests. we'll work on testing images and reasoning effort etc running constantly as chris mentions in point 3.

(screenshot showing our internal testing of deepinfra image inputs: https://raw.githubusercontent.com/ping-Toven/images/main/ima... )

Re: So you want to use OpenRouter?

#113

Post by numlocked »

frenchtoast8 wrote: post #102
numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

What are your plans to improve a substantial lack of customer support? https://news.ycombinator.com/item?id=49577159

Hmm, let me check. That certainly seems wrong. Can you send me an email w/ your email so I can look into it? im cc at openrouter.ai. Or DM me on X? x.com/cclark

Re: So you want to use OpenRouter?

#116

Post by Aurornis »

joshstrange wrote: post #20

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “Ope…

I treat OpenRouter as a central point to access specific providers. Pinning is a given for that use case.

I’ve tried using un-pinned models and the experience is exactly as you described: Some providers are so unreliable that the majority of requests fail. Some providers do weird things like abruptly end the response (which I get billed for and have to re-submit). Some providers are clearly running heavily quantized versions of the model because their eval performance is terrible. Some providers advertise features on OpenRouter but will reject those requests when submitted to their API.

So pinning is the way to go.

Re: So you want to use OpenRouter?

#117

Post by ActivePattern »

joelthelion wrote: post #64
vova_hn2 wrote: post #38

> The same model will benchmark very differently Providers probably serve quantized versions without disclosing it. Which is a real shame, because for certain tasks I would be perfectly willing to trade accuracy for cost. But, unfortunately, it is impossible to explicitly choose how quantized do you want your model to be, unless you are running it yourself on your own (or rented) hardware. BTW, does anyone knows if L…

> Providers probably serve quantized versions without disclosing it It should be OpenRouter's responsibility to protect you against it, by regularly benchmarking providers and giving you the control to avoid bad providers. In fact, that's a big opportunity for them, since it justifies their place as a middleman between users and inference providers.

100% agree. The key issue is that users think they are getting results from a specific model configuration and they are clearly not, which is fraudulent.

If OpenRouter wants to succeed as a business, they need to be auditing the providers they connect to (i.e. benchmarking) and removing fraudulent ones from their service.

Re: So you want to use OpenRouter?

#118

Post by random3 »

joshstrange wrote: post #20

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “Ope…

This (by definition) the reality of every normalization/proxy layer, standard, etc. There’s always a cost for flexibility/freedom. What’s traded is when and by whom it’s paid.

Re: So you want to use OpenRouter?

#119

Post by bradfa »

bambax wrote: post #62

Very informative article, but I'd say all of this is in favor of OpenRouter, not against it. It gives you full flexibility, not just of models, but of providers. That's the offer!

As long as you are constantly diligent in staying on top of what each provider is doing that may impact you.

It reads to me an argument for self hosting, maybe a less capable model to deal with smaller compute resources, but when starting to use that model and inference software to build a benchmark you can easily rerun when you change things to observe the impact of the change.

Maybe that’s a similar level of diligence but they feel different to me.

Re: So you want to use OpenRouter?

#120

Post by frenchtoast8 »

numlocked wrote: post #113
frenchtoast8 wrote: post #102

Earlier quoted context omitted.

What are your plans to improve a substantial lack of customer support? https://news.ycombinator.com/item?id=49577159

Hmm, let me check. That certainly seems wrong. Can you send me an email w/ your email so I can look into it? im cc at openrouter.ai. Or DM me on X? x.com/cclark

My ticket number is #107156 (opened 9 days ago). The point is that the Discord is full of people not getting responses, and I'm not even the one waiting the longest. The only help they get is from the AI assistant in the server saying don't bump the ticket or you'll be punished by getting moved to the back of the queue. So there's nothing anybody can do but wait who knows how long.
Post reply on HN