Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to use OpenRouter?

mmoustafa.com

Post reply on HN
Print view View original

81–90 of 208 posts

Re: So you want to use OpenRouter?

#81

Post by epistasis »

LLM ops is not trivial. The systems for running inference are very complex, and running across multiple GPUs and nodes adds tons more complexity. And when LLMs are run incorrectly, they still work, just not at optimal performance. Even noticing that something is wrong is not trivial, and finding the problem is far far harder.

So I guess I shouldn't be surprised at all to see these benchmarks, but still I am!

There are such huge economies of scale with batched inference that it's clear this sort of service will continue, but it has a lot of growing up to do. Even AWS Bedrock has a Claude that feels different to me, but I haven't had a chance to do actual benchmarks that would show that.

Re: So you want to use OpenRouter?

#82

Post by copperx »

Macha wrote: post #32

I do use OpenRouter for my personal use and this matches with my experience. I still use it because of it's top up model providing a way to not get surprised by out of control costs and being able to switch models with one account, but it's very noticeable at times. The models I've mainly been using recently are GLM 5.3 Flash and GLM 5.3. While obviously all these models have some variability, GLM 5.3 Flash feels lik…

Using the same provider?

Re: So you want to use OpenRouter?

#83

Post by FranklinMaillot »

I've had the same experience in my personal use. Unreliable output, thinking token leaking into the conversation... Time to first token and tk/s also vary wildly per provider.

The worst case of hallucination I had, was DS v4 flash switching to Italian mid-conversation and impersonating a podcast host for no reason.

Re: So you want to use OpenRouter?

#84

Post by porridgeraisin »

embedding-shape wrote: post #43
bbor wrote: post #41

Earlier quoted context omitted.

if you want to control the routing, you'd lock down providers I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled…

> I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. Why do you care about the public benchmarks at all? The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use…

You'd be surprised. Not enough teams still have their own evals. India[1] and the US[2] is my experience. To get many of them to understand the benefit of putting a couple of people to do data labelling and write a couple of verifiers for just a few days every few months was so difficult. Many folks have understood it all wrong and made allotments like "big model for this task" "small model for this task". Small was sometimes parameters, sometimes brand version number, or sometimes because it has "mini" in its name.

Proper LLM adoption beyond fucking around with claude code or github copilot is so low and is only going to go up as people figure it out. I also think the new cloud agents thing might accelerate adoption among these companies, since it's a bit more plug and play. But they are more likely to be stingy about it, so I'm not sure about the high margin expectations of certain model families.

[1] not witch, but BFSI. Surprisingly parts of witch companies have it down to science already, they embedded openai or anthropic or startups like devrev more than a year back.

[2] again not high fly SF companies, BFSI.

Re: So you want to use OpenRouter?

#85

Post by sinuhe69 »

"OpenRouter runs per-provider benchmarks on the same model: GPQA Diamond and TAU-Bench Airline (a tool-calling task)." why I haven't never seen it? Click on the link brings nothing! Benchmark is only for the model. Per provider is a performance matrix (latency, throughput) What did I miss?

--

Update: oh, that is the AutoExacto Benchmarks! Now I see it.

Re: So you want to use OpenRouter?

#86

Post by nojs »

This is an issue self hosting as well. There’s a lot of footguns that give you slightly bad results.

I wonder what tricks one could use to ensure the model is actually performing on par with the reference api, like matching seeds or running exact benchmarks.

Re: So you want to use OpenRouter?

#87

Post by numlocked »

bbor wrote: post #36

Wait WTF?! I thought performance was, well, performance, not efficacy ! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I have no idea where the author got those clean charts. "AutoExacto" numbers (completely meaningless name) are available for 6 providers, two of which are the same provider, and none of which are even in the top half throughput-wise. When you click "+28 more pr…

We run benchmarks against all of our endpoints, in production. That first chart that the author shows is in fact our live benchmarking data. If providers underperform, we kick them out of the routing pool. That is why we run those benchmarks. Performance

And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of valuable work! https://openrouter.ai/docs/guides/routing/auto-exacto

Re: So you want to use OpenRouter?

#88

Post by artursapek »

joshstrange wrote: post #20

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “Ope…

I only use OpenRouter to benchmark models. For my production app, I have direct integrations with OpenAI, Anthropic, Google, xAI. Even just input caching is enough of a reason to do that, assuming you are trying to build the fastest and most cost-efficient product possible. I don't understand how people supposedly run production apps using OpenRouter unless those two variables somehow don't matter to them.

Re: So you want to use OpenRouter?

#89

Post by dannyw »

epistasis wrote: post #81

LLM ops is not trivial. The systems for running inference are very complex, and running across multiple GPUs and nodes adds tons more complexity. And when LLMs are run incorrectly, they still work, just not at optimal performance. Even noticing that something is wrong is not trivial, and finding the problem is far far harder. So I guess I shouldn't be surprised at all to see these benchmarks, but still I am! There ar…

Bedrock Claude is absolutely not identical to 1P Claude.

They’re close enough to not matter though.

Re: So you want to use OpenRouter?

#90

Post by dannyw »

JaceComix wrote: post #78
bsaul wrote: post #5

That looks absolutely horrifying. What are the alternatives ??

Fireworks hosts the available models themselves which probably solves the problem consistency problem that OP had to deal with. It's been a few months since I looked around at this topic, but Fireworks and Openrouter were the two options I (briefly) tried.

Fireworks has high variance amongst models and while some are served correctly; many of them are junk / broken and degraded and it seems like they don’t even know; because even running 1k MMLU Pro questions would flag it very quickly.
Post reply on HN