Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

401–410 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#401

Post by Den_VR »

chillee wrote: post #395

This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it's 1.44 million input tokens per second * 37 billion active params * 2 (FMA) / 8 [GPUs per instance] = 13 Petaflops per second. That's approximately 7x absolutely peak FLOPS on the hardware. Obviously, that's impossible. There's many other…

So, bottom line, do you think it’s probable that either OpenAI or Anthropic are “losing money on inference?”

Re: Are OpenAI and Anthropic losing money on inference?

#402

Post by dcre »

jrflowers wrote: post #394
dcre wrote: post #366

Earlier quoted context omitted.

Zitron is not a serious analyst. https://bsky.app/profile/davidcrespo.bsky.social/post/3lxale... https://bsky.app/profile/davidcrespo.bsky.social/post/3lo22k... https://bsky.app/profile/davidcrespo.bsky.social/post/3lwhhz... https://bsky.app/profile/davidcrespo.bsky.social/post/3lv2dx...

Ed Zitron: I don’t think OpenAI will become profitable The link you posted: I think it is very plausible that it will be hard for OpenAI to become profitable

Are you referring to the post where I listed 4 claims and marked one ridiculous, one wrong, one unlikely, and one plausible?

He is not wrong about everything. For example, after Sam Altman said in January that OpenAI would introduce a model picker, Zitron was able to predict in March that OpenAI would introduce a model picker. And he was right about that.

Re: Are OpenAI and Anthropic losing money on inference?

#403

Post by pas »

jacurtis wrote: post #398
BlindEyeHalo wrote: post #303

Earlier quoted context omitted.

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

In a recent episode of Hard Fork podcast, the hosts discussed an on-the-record conversation they had with Sam Altman from OpenAI. They asked him about profitability and he claimed that they are losing money mostly because of the cost of training. But as the model advances, they will train less and less. Once you take training out of the equation he claimed they were profitable based on the cost of serving the trained…

> most AI companies are losing money right now

which is completely "normal" at this point, """right"""? if you have billions of VC money chasing returns there's no time to sit around, it's all in, the hype train doesn't wait for bootstrapping profitability. and of course with these gargantuan valuations and mandatory YoY growth numbers, there is no way they are not fucking with the unit economy numbers too. (biases are hard to beat, especially if there's not much conscious effort to do so.)

Re: Are OpenAI and Anthropic losing money on inference?

#404

Post by ascorbic »

topaz0 wrote: post #392
ascorbic wrote: post #241

Earlier quoted context omitted.

This article is about the model providers' costs, not API users'. Cursor etc have to pay the marked-up inference costs, so it's not surprising they can't make a profit.

Yes, and the comment you first replied to was about the state/viability of the industry as a whole. If users can't make money from this "transformative technology", even when the provider is in the stage of burning money for the sake of growth, that sort of tells against it turning into a trillion dollar industry or whatever the hype claims.

The point is that the providers aren't burning money by subsidising inference costs. On the contrary, if this article is to believed they're charging healthy margins on it.

So there are two answers: for the model providers, it's because they're spending it all on training the next model. For the API users, it's because they're spending it all on expensive API usage.

Re: Are OpenAI and Anthropic losing money on inference?

#405

Post by chillee »

Den_VR wrote: post #401
chillee wrote: post #395

This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it's 1.44 million input tokens per second * 37 billion active params * 2 (FMA) / 8 [GPUs per instance] = 13 Petaflops per second. That's approximately 7x absolutely peak FLOPS on the hardware. Obviously, that's impossible. There's many other…

So, bottom line, do you think it’s probable that either OpenAI or Anthropic are “losing money on inference?”

No. In some sense, the article comes to the right conclusion haha. But it's probably >100x off on its central premise about output tokens costing more than input.

Re: Are OpenAI and Anthropic losing money on inference?

#406

Post by doctorpangloss »

chillee wrote: post #405
Den_VR wrote: post #401

Earlier quoted context omitted.

So, bottom line, do you think it’s probable that either OpenAI or Anthropic are “losing money on inference?”

No. In some sense, the article comes to the right conclusion haha. But it's probably >100x off on its central premise about output tokens costing more than input.

I’m pretty sure input tokens are cheap because they want to ingest the data for training later no? They want huge contexts to slice up.

Re: Are OpenAI and Anthropic losing money on inference?

#407

Post by pama »

chillee wrote: post #395

This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it's 1.44 million input tokens per second * 37 billion active params * 2 (FMA) / 8 [GPUs per instance] = 13 Petaflops per second. That's approximately 7x absolutely peak FLOPS on the hardware. Obviously, that's impossible. There's many other…

Agree that the writeup is very wrong, especially for the output tokens. Here is how anyone with enough money to allocate a small cluster of powerful GPUs can decode huge models at scale, since nearly 4 months ago, with costs of 0.2 USD/million output tokens.

https://lmsys.org/blog/2025-05-05-large-scale-ep/

This has gotten significantly cheaper yet with additional code hacks since then, and with using the B200s.

Re: Are OpenAI and Anthropic losing money on inference?

#408

Post by atq2119 »

fallmonkey wrote: post #308

The estimation for output token is too low since one reasoning-enabled response can burn through thousands of output tokens. Also low for input tokens since in actual use there're many context (memory, agents.md, rules, etc) included nowadays.

When using APIs, you pay for reasoning tokens like you do for actual outputs. So, the estimation on a per-token basis is not affected by reasoning.

What reasoning affects is the ratio of input to output tokens, and since input tokens are cheaper, that may well affect the economics in the end.

Re: Are OpenAI and Anthropic losing money on inference?

#409

Post by martinald »

chillee wrote: post #405
Den_VR wrote: post #401

Earlier quoted context omitted.

So, bottom line, do you think it’s probable that either OpenAI or Anthropic are “losing money on inference?”

No. In some sense, the article comes to the right conclusion haha. But it's probably >100x off on its central premise about output tokens costing more than input.

Thanks for the correction (author here). I'll update the article - very fair point on compute on input tokens which I messed up. Tbh I'm pleased my napkin math was only 7x off the laws of physics :).

Even rerunning the math on my use cases with way higher input token cost doesn't change much though.

Re: Are OpenAI and Anthropic losing money on inference?

#410

Post by [deleted] »

chillee wrote: post #395

This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it's 1.44 million input tokens per second * 37 billion active params * 2 (FMA) / 8 [GPUs per instance] = 13 Petaflops per second. That's approximately 7x absolutely peak FLOPS on the hardware. Obviously, that's impossible. There's many other…

[deleted]
Post reply on HN