121–130 of 208 posts
If you for some reason had unusee balance and forget about it; its just gone.
Earlier quoted context omitted.
Fireworks hosts the available models themselves which probably solves the problem consistency problem that OP had to deal with. It's been a few months since I looked around at this topic, but Fireworks and Openrouter were the two options I (briefly) tried.
Fireworks has high variance amongst models and while some are served correctly; many of them are junk / broken and degraded and it seems like they don’t even know; because even running 1k MMLU Pro questions would flag it very quickly.
Source: Enterprise customer doing $XXM annual run rate of inference spend on their platform
Open router have one more annoying disadvantage: credits you buy expire in around 1 year +/- ~30 days. They do sent warning email 30 days in advance, but then just make them disappear. If you for some reason had unusee balance and forget about it; its just gone.
Yeah, it is actually very bad now. And somehow keeps leaking keys.
This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “Ope…
If I could pay per request without maintaining a balance or credit card out of a single wallet (using crypto or something maybe) I would happily simply write the integration myself because OR’s caching is often not as good without some hoop jumping.
I have been working on a product for months and for 1 of the specific models have become the number 3 user and very likely soon the number 2. Its a google model. The edge cases are crazy to deal with and have taken a long time to find. I am also finding that since it has no fallbacks but no published rate limit I am single handedly taking the model down on what I thought were a reasonable amount of request. There is…
Earlier quoted context omitted.
1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them. 2. As long as the apis use the openai spec, it should be fine? And what makes you say the providers are using different weights? The blogpost says the exact opposite? 3. Great example, if amazon does not lead…
> 1. Yes, but if i can't rely on openrouter to route to providers to give me the best price and the best uptime across a number of providers, and it instead gives me inconsistent results, then I will not use them. OpenRouter does reliably route to your specified model and provider, otherwise it'd pretty much be fully broken. Parent is complaining about the auto-provider chosing, not that all providers are unreliable.…
Earlier quoted context omitted.
I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible. For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.
Did you miss the part of this article where the benchmarks specifically call out Fireworks as one of the worst in their tests: > Fireworks scored 46% on TAU, a 30 point gap Another surprise was DigitalOcean being bottom of barrel too. Companies are apparently willing to risk their brand name by being deceptive about these heavily quantized/flawed model-serving.