Are OpenAI and Anthropic losing money on inference?

Post by ozgung » Thu, Aug 28, 2025, 3:47 PM UTC

GaggiX wrote: post #106

ekelsen wrote: post #98

The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on…

Your calculations make no sense. Why are you loading the model for each token independently? You can process all the input tokens at the same time as long as they can fit in memory. You are doing the calculation as they were output tokens on a single batch, it would not make sense even in the decode phase.

Post by scrollaway » Thu, Aug 28, 2025, 3:48 PM UTC

martinald wrote: post #163

scrollaway wrote: post #76

Earlier quoted context omitted.

> claude code was completely unsustainable and therefore a flash in the pan and devs aren't at risk How can you possibly say this if you know anything about the evolution of costs in the past year? Inference costs are going down constantly, and as models get better they make less mistakes which means less cycles = less inference to actually subsidize. This is without even looking at potential fundamental improvements…

I totally agree with you! I have heard others saying this though. But I don't think it's true.

Post by JSR_FDED » Thu, Aug 28, 2025, 3:50 PM UTC

churchill wrote: post #104

wongarsu wrote: post #72

Earlier quoted context omitted.

But they don't have to be retained frequently at great cost. Right now they are retrained frequently because everyone is frequently coming out with new models and nobody wants to fall behind. But if investment for AI were to dry up everyone would stop throwing so much money at R&D, and if everyone else isn't investing in new models you don't have to either. The models are powerful as they are, most of the knowledge i…

Like we've seen with Karparthy & Murati starting their own labs, it's to be expected that over the next 5 years, hundreds of engineers & researchers at the bleeding edge will quit and start competing products. They'll reliably raise $1b to $5b in weeks, too. And it's logical: for an investor, a startup founded by a Tier 1 researcher will more reliably 10-100x your capital, vs. Anthropic & OpenAI that are already at >…

Post by dragonwriter » Thu, Aug 28, 2025, 3:52 PM UTC

noodletheworld wrote: post #145

Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap,…

Post by empath75 » Thu, Aug 28, 2025, 3:53 PM UTC

hirako2000 wrote: post #164

noodletheworld wrote: post #145

Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap,…

Imo the article is totally off the mark since it assumes users on average do not go over th 1M tokens per day. Afaik openai doesn't enforce a daily quota even on the $20 plans unless the platform is under pressure. Since I often consume 20M token per day, one can assume many would use far more than the 1M tokens assumed in the article's calculations.