Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to use OpenRouter?

mmoustafa.com

Post reply on HN
Print view View original

191–200 of 208 posts

Re: So you want to use OpenRouter?

#191

Post by Computer0 »

numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

I switched to Opencode because they have guaranteed performance. Maybe you could have an openrouter certified badge or label for this sort of thing.

Re: So you want to use OpenRouter?

#192

Post by sheo »

plandis wrote: post #187

> The same model will benchmark very differently This is surprising to me. Does anyone have a definitive answer that accounts for this difference between providers? Do the benchmarks that Open Router runs not account for the stochastic nature of the models? Do the providers lie about quantization or context window sizing? Does Open Router not take into account variations for a given model? I could understand latency/…

As said in the article, all of the providers run different runtimes with proprietary, sometimes untested and weird optimisations

There are multiple types of quantisation: weights and KV Cache. Quantizing kv cache can drastically hurt performance

Re: So you want to use OpenRouter?

#193

Post by [deleted] »

numlocked wrote: post #79

Co-founder and COO of OpenRouter here. Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically). Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another: 1. Let you use a lot of capacity across a lot of providers, in a w…

[deleted]

Re: So you want to use OpenRouter?

#194

Post by fzysingularity »

msp26 wrote: post #163
fzysingularity wrote: post #111

The author’s comments on vision providers is especially interesting. We saw that most providers don’t provide native video url support, have high-variability in vision performance (likely due to the fact that they’re serving different quantization levels behind the same model id). If you’re building vision-native apps, there are so many footguns in vLLM/SGLang serving configurations, let alone the routing/orchestrati…

Do you have any advice on this front? I use vllm for a project but only for text tasks at the moment.

text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want it to do fine-grained localization (time or spatial), and the vllm default params can be way off for your use-case.

For example, most vision models don't need 256K context length for modes like Qwen3.8-27B when all you care about is single-image captioning, so you can technically save on KV cache. Video reasoning does require that context length, so it's a different set of deployment parameters that need to be enabled.

All of this to say that the providers that offer these models, are simply using vLLM / SGLang, and mostly cater to the text inference use-case (coding, etc). Vision always seems to be a bit of an afterthought.

Re: So you want to use OpenRouter?

#195

Post by jeremyjh »

TZubiri wrote: post #155
mrngld wrote: post #48

Earlier quoted context omitted.

How does it erase the notion of competition driving down prices? Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Don't load 6 months' worth of credit in an account, keep it tight. There's fairly little lock-in. The most significant lock-in to me isn't even something you mentioned, but rather it's model related; I personally put a little time into trying to optim…

>Endpoints are largely compatible, so the code change required to switch from one to another is trivial. Wrapping a specific implementation in a neutral function is something you learn to do in year 1 of programming. This specific issue and argument I see in lots of different aggregator dependencies, Terraform, LiteLLM/OpenRouter. They promise to save some hypothetical work in the future if your boss asks to change v…

I think OpenRouter’s value proposition is less about avoiding two hour developer tasks and more about having a single place to establish policy controls and dynamic selection based on current pricing and performance data. If you can’t actually do that - for the reasons described in the article you end up pinning - they can’t deliver. But it seems to work for some use cases.

Re: So you want to use OpenRouter?

#196

Post by SXX »

numlocked wrote: post #170
radicality wrote: post #124

Earlier quoted context omitted.

Oh, interesting, I feel like a year so ago I read something posted from openrouter team that they write that credits expire, but that they in actuality don’t expire them. But maybe I’m misremembering, or perhaps that’s changed for the worse in last few months :S

We started actually expiring the credits a ~month ago. If you make any kind of API request, it resets the clock. We try to make it a very generous policy, but we can't keep a monotonically increasing liability on the books. We end up owing (a lot) of taxes on it, but can't actually recognize revenue. We would much rather you spend the credits! Hence the reminder emails, and generous "clock reset" policy.

Is this reset clock policy is actually documented somewhere? Because right now your website it's just say 1 year since credit purchase.

Re: So you want to use OpenRouter?

#197

Post by SXX »

wren6991 wrote: post #178
numlocked wrote: post #170

Earlier quoted context omitted.

We started actually expiring the credits a ~month ago. If you make any kind of API request, it resets the clock. We try to make it a very generous policy, but we can't keep a monotonically increasing liability on the books. We end up owing (a lot) of taxes on it, but can't actually recognize revenue. We would much rather you spend the credits! Hence the reminder emails, and generous "clock reset" policy.

> but can't actually recognize revenue Hmm? You have the revenue already. I know it's awkward from an accounting point of view, but you already took my money. "Letting" me keep the balance in the account is not generous. Edit: on re-reading this came out more combative than I intended, sorry. I think what you're doing is reasonable.

I kind a get their reasoning. They cant recognize balance topup as revenue because then they'll owe taxes on full amount.

At the same time majority of money we pay them usually just gonna be an expense paid to actual inference providers. Then they pay taxes on their fee aka actual profits.

Its understandable, but it dont make 1 year expiration any good.

Re: So you want to use OpenRouter?

#198

Post by aftbit »

The big advantage of OpenRouter IMO is that you don't have to manage keys, URLs, and subscriptions with 10+ providers. You definitely should still pick providers and pin them, at least if you care about repeatable results.

Re: So you want to use OpenRouter?

#199

Post by msp26 »

fzysingularity wrote: post #194
msp26 wrote: post #163

Earlier quoted context omitted.

Do you have any advice on this front? I use vllm for a project but only for text tasks at the moment.

text is mostly beaten to death, so you can expect good defaults to work for vllm. VLMs specifically are quite sensitive to quantization, especially if you want it to do fine-grained localization (time or spatial), and the vllm default params can be way off for your use-case. For example, most vision models don't need 256K context length for modes like Qwen3.8-27B when all you care about is single-image captioning, so…

Got it, I'll test more deployment params when I benchmark models for vision tasks.

Do you have any tips for Gemma 4 31B in particular? I quite like the model but I feel like I'm underutilizing my rented GPU hard due to skill issues. Throughput should be way higher than this, right?

---

Gemma 4 31B NVFP4, vLLM 0.29, single B200 (modal), FlashInfer, fp8 KV, prefix caching, 32k ctx

Workload: ~6k-token shared prefix (~98% cache hit) + short input, ~250 tokens out, ~250 seqs running. Get ~5-6k output tok/s, flat from 128 to 384 concurrency.

Spec decode (DFlash, n-gram) didn't give a good boost which is annoying because I feel like this task should be easy for a draft model to predict.

Individual request latency doesn't matter I just need as much throughput as possible. Its only active for a few hours when I need it.

Re: So you want to use OpenRouter?

#200

Post by polytely »

I've just been using openrouter to be able to quickly switch models and keep tight control over how much i spend and for that it's great. i also love their chat thing on the site where you can ask multiple models simultaneously , very good for brainstorming
Post reply on HN