Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you want to use OpenRouter?

mmoustafa.com

Post reply on HN
Print view View original

51–60 of 208 posts

Re: So you want to use OpenRouter?

#51

Post by embedding-shape »

bbor wrote: post #47
embedding-shape wrote: post #43

Earlier quoted context omitted.

> I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. Why do you care about the public benchmarks at all? The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use…

If openrouter is only for people who "make their own benchmark" in a mission to roll something that's usually made by scientists with large budgets, it should say so and thus fade into deserved obscurity. I don't think anyone serious is just willy-nilly making individual requests against OpenRouter Despite your confidence, that is indeed the basis of this massive corporations entire business plan. If you just want be…

[flagged]

Re: So you want to use OpenRouter?

#52

Post by miki123211 »

joshstrange wrote: post #20

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “Ope…

I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible.

For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.

Re: So you want to use OpenRouter?

#53

Post by nacs »

miki123211 wrote: post #52
joshstrange wrote: post #20

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “Ope…

I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible. For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.

Did you miss the part of this article where the benchmarks specifically call out Fireworks as one of the worst in their tests:

> Fireworks scored 46% on TAU, a 30 point gap

Another surprise was DigitalOcean being bottom of barrel too.

Companies are apparently willing to risk their brand name by being deceptive about these heavily quantized/flawed model-serving.

Re: So you want to use OpenRouter?

#54

Post by nacs »

danvdb wrote: post #40
bsaul wrote: post #5

That looks absolutely horrifying. What are the alternatives ??

I've used Requesty ( https://www.requesty.ai ), let's you pin down providers and build your own routing policy so you at least somewhat know what to expect.

Openrouter lets you pin or blacklist providers or specify provider per-request as well.

Re: So you want to use OpenRouter?

#55

Post by nhecker »

CubsFan1060 wrote: post #15

Another thing to be aware of -- apparently their billing limits don't work. Somehow someone was able to use my key from Singapore. It had a $10 daily limit and they were able to spend $100. Not only that, it got me blocked from all models for the frontier labs. On top of that, near as I can tell, there are no protections for your API key. No restrictions by country, IP, etc...

Just chiming in that I've never had this experience with them. I set, hit, and depend on those limits regularly. I would be wildly interested in reading what their support analysis had to say about this situation, if such a postmortem was done.

Re: So you want to use OpenRouter?

#57

Post by neya »

The best part about OpenRouter is 200 OK is probably hardcoded into their responses.

I used to get content: "" all the time and I used to triple check my code to see if I was doing something wrong until I realized most AI providers in general have vibe coded their infrastructure as well and it is just a futile attempt to even fight it.

Post reply on HN