Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

241–250 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#241

Post by ascorbic »

topaz0 wrote: post #103
ascorbic wrote: post #27

Earlier quoted context omitted.

Because they're spending it all on training the next model.

That's an argument for why openai and anthropic shouldn't be profitable, but this point is about how also they don't have customers using the models to generate a profit either. Things like cursor, for example. ETA: also note the recent MIT study that found that 95% of LLM pilots at for-profit companies were not producing returns.

This article is about the model providers' costs, not API users'. Cursor etc have to pay the marked-up inference costs, so it's not surprising they can't make a profit.

Re: Are OpenAI and Anthropic losing money on inference?

#242

Post by NoahZuniga »

ankit219 wrote: post #236
thegeomaster wrote: post #216

Earlier quoted context omitted.

Are you saying that you think Sonnet 4 has 100B-200B _active_ params? And that Opus has 2T active? What data are you basing these outlandish assumptions on?

Oh nothing official. There are people who estimate the sizes based on tok/s, cost, benchmarks etc. The one that most go on is https://lifearchitect.substack.com/p/the-memo-special-editio... . This guy estimated Claude 3 opus to be 2T param model (given the pricing + speed). Opus 4 is 1.2T param according to him (but then I dont understand why the price remained the same.). Sonnet is estimated by various people to be…

If you're using the api cost of the model to estimate it's size, then you can't use this size estimate to estimate the inference cost.

Re: Are OpenAI and Anthropic losing money on inference?

#243

Post by NoahZuniga »

eikenberry wrote: post #204
Aurornis wrote: post #64

Earlier quoted context omitted.

Exactly. All of the claims that OpenAI is losing money on every request are wrong. OpenAI hasn’t even unlocked all of their possible revenue opportunities from the free tier such as ads (like Google search), affiliate links, and other services. There’s also a lot of comments in this thread who want LLM companies to fail for different reasons, so they’re projecting that wish on to imagined unit economics. I’m having f…

So inference is cheap but training is expensive and getting more expensive. It seems like if they can't get training expenses down, cheap inference won't matter.

No. Training itself isn't that expensive compared to inference. The real expense is salary for talent.

Re: Are OpenAI and Anthropic losing money on inference?

#244

Post by overgard »

I thought the thing that made DeepSeek interesting (besides competition from China) was that its inference costs were something like 1/10th. So unless that gap has been bridged (has it?) I don't think a calculation based on DeepSeek can apply to OpenAI or Anthropic.

Re: Are OpenAI and Anthropic losing money on inference?

#245

Post by overgard »

thatguysaguy wrote: post #201
sc68cal wrote: post #125

This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.

Why would you think that deepseek is more efficient than gpt-5/Claude 4 though? There's been enough time to integrate the lessons from deepseek.

Because to make GPT-5 or Claude better than previous models, you need to do more reasoning which burns a lot more tokens. So, your per-token costs may drop, but you may also need a lot more tokens.

Re: Are OpenAI and Anthropic losing money on inference?

#246

Post by qrios »

Onavo wrote: post #227
qrios wrote: post #203

For sure an interesting calculation. Only one remark from someone with GPU metal experience: > But compute becomes the bottleneck in certain scenarios. With long context sequences, attention computation scales quadratically with sequence length. Even if the statement about quadratically scales is right, the bottleneck we are talking about is somewhere north by factor 1000. If 10k cores do only simple matrix operation…

What do you mean by collision?

If multiple cores tries to get the same memory addresses, the MMU feeds only one core, the second one have to whait. Depends on the type of RAM, this will cost a lot of cycles.

GPU MMUs can handle multiple line in parallel. But not 10k cores at the same time. The HBM is not able to transfer 3.5TByte sequencial.

Re: Are OpenAI and Anthropic losing money on inference?

#247

Post by 827a »

simonw wrote: post #213

https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chat... quotes Sam Altman saying: > Most of what we're building out at this point is the inference [...] We're profitable on inference. If we didn't pay for training, we'd be a very profitable company.

From the latest NYT Hard Fork podcast [1]. The hosts were invited to a dinner hosted by Sam, where Sam said "we're profitable if we remove training from the equation", they report he turned to Lightcap (COO) and asked "right?" and Lightcap gave an "eeekk we're close".

They aren't yet profitable even just on inference, and its possible Sam didn't know that until very recently.

[1] https://www.nytimes.com/2025/08/22/podcasts/is-this-an-ai-bu...

Re: Are OpenAI and Anthropic losing money on inference?

#248

Post by overgard »

dcre wrote: post #218
sc68cal wrote: post #125

This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.

What are we meant to take away from the 8000 word Zitron post? In any case, here is what Anthropic CEO Dario Amodei said about DeepSeek: "DeepSeek produced a model close to the performance of US models 7-10 months older, for a good deal less cost (but not anywhere near the ratios people have suggested)" "DeepSeek-V3 is not a unique breakthrough or something that fundamentally changes the economics of LLM’s; it’s an e…

In terms of sources, I would trust Zitron a lot more than Altman or Amodei. To be charitable, those CEOs are known for their hyperbole and for saying whatever is convenient in the moment, but they certainly aren't that careful about being precise or leaving out inconvenient details. Which is what a CEO should do, more or less, but, I wouldn't trust their word on most things.

Re: Are OpenAI and Anthropic losing money on inference?

#249

Post by KallDrexx »

Since DeepSeek R1 is open weight, wouldn't it be better to validate the napkin math to validate how many realistic LLM full inferences can be done on a single H100 in a time period, and calculate the token cost of that?

Without having in depth knowledge of the industry, the margin difference between input and output tokens is very odd to me between your napkin math and the R1 prices. That's very important as any reasoning model explodes reasoning tokens, which means you'll encounter a lot more output tokens for fewer input tokens, and that's going to heavily cut into the high margin ("essentially free") input token cost profit.

Unless I'm reading the article wrong.

Re: Are OpenAI and Anthropic losing money on inference?

#250

Post by selimthegrim »

dcre wrote: post #219
simonw wrote: post #213

https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chat... quotes Sam Altman saying: > Most of what we're building out at this point is the inference [...] We're profitable on inference. If we didn't pay for training, we'd be a very profitable company.

ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…

This sounds like fabs.
Post reply on HN