Are OpenAI and Anthropic losing money on inference? • Hacker News Board

Re: Are OpenAI and Anthropic losing money on inference?

#231

Post by gmerc » Thu, Aug 28, 2025, 4:43 PM UTC

dcre wrote: post #218

sc68cal wrote: post #125

This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.

What are we meant to take away from the 8000 word Zitron post? In any case, here is what Anthropic CEO Dario Amodei said about DeepSeek: "DeepSeek produced a model close to the performance of US models 7-10 months older, for a good deal less cost (but not anywhere near the ratios people have suggested)" "DeepSeek-V3 is not a unique breakthrough or something that fundamentally changes the economics of LLM’s; it’s an e…

Grok 3.5: 400M training run DeepSeek R1: 5M training run Released around the same time, marginal performance difference.

Re: Are OpenAI and Anthropic losing money on inference?

#232

Post by ugh123 » Thu, Aug 28, 2025, 4:47 PM UTC

asabla wrote: post #225

ugh123 wrote: post #220

Earlier quoted context omitted.

That might be the case, but inference times have only gone up since GPT-3 (GPT-5 is regularly 20+ seconds for me).

And by GPT-5 you mean through their API? Directly through Azure OpenAI services? or are you talking about ChatGPT set to using GPT-5. All of these alternatives means different things when you say it takes +20 seconds for a full response.

Sure, apologies. I mean ChatGPT UI

Re: Are OpenAI and Anthropic losing money on inference?

#233

Post by fancyfredbot » Thu, Aug 28, 2025, 4:50 PM UTC

When you are operating at scale you are likely to use a small model during the auto regressive phase to generate sequential tokens and only involve the large model once you've generated several tokens. Whenever the two predict the same output you effectively generate more than one token at a time. The idea is the models will agree often enough to significantly reduce output token costs. Does anyone know how effective that is in practice?

Re: Are OpenAI and Anthropic losing money on inference?

#234

Post by prasadjoglekar » Thu, Aug 28, 2025, 4:51 PM UTC

dcre wrote: post #229

drob518 wrote: post #223

Earlier quoted context omitted.

Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and therefore basically meaningless.

The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.

Well, only if the one training model continued to function as a going business. Their amortization window for the training cost is 2 months or so. They can't just keep that up and collect $.

They have to build the next model, or else people will go to someone else.

Re: Are OpenAI and Anthropic losing money on inference?

#235

Post by ToucanLoucan » Thu, Aug 28, 2025, 4:53 PM UTC

dcre wrote: post #229

drob518 wrote: post #223

Earlier quoted context omitted.

Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and therefore basically meaningless.

The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.

So is OpenAI capable of not making a new model at some point? They've been training the next model continuously as long as they've existed AFAIK.

Our software house spends a lot on R&D sure, but we're still incredibly profitable all the same. If OpenAI is in a position where they effectively have to stop iterating the product to be profitable, I wouldn't call that a very good place to be when you're on the verge of having several hundred billion in debt.

Re: Are OpenAI and Anthropic losing money on inference?

#236

Post by ankit219 » Thu, Aug 28, 2025, 4:53 PM UTC

thegeomaster wrote: post #216

ankit219 wrote: post #160

This seems very very far off. From the latest reports, anthropic has a gross margin of 60%. It came out in their latest fundraising story. From that one The Information report, it estimated OpenAI's GM to be 50% including free users. These are gross margins so any amortization or model training cost would likely come after this. Then, today almost every lab uses methods like speculative decoding and caching which red…

Are you saying that you think Sonnet 4 has 100B-200B _active_ params? And that Opus has 2T active? What data are you basing these outlandish assumptions on?

Oh nothing official. There are people who estimate the sizes based on tok/s, cost, benchmarks etc. The one that most go on is https://lifearchitect.substack.com/p/the-memo-special-editio.... This guy estimated Claude 3 opus to be 2T param model (given the pricing + speed). Opus 4 is 1.2T param according to him (but then I dont understand why the price remained the same.). Sonnet is estimated by various people to be around 100B-200B params.

[1]: https://docs.google.com/spreadsheets/d/1kc262HZSMAWI6FVsh0zJ...

Re: Are OpenAI and Anthropic losing money on inference?

#237

Post by gomox » Thu, Aug 28, 2025, 4:55 PM UTC

drob518 wrote: post #223

simonw wrote: post #213

https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chat... quotes Sam Altman saying: > Most of what we're building out at this point is the inference [...] We're profitable on inference. If we didn't pay for training, we'd be a very profitable company.

Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and therefore basically meaningless.

I can't imagine the hoops an accountant would have to go through to argue training cost is COGS. In the most obvious stick-figures-for-beginners interpretation, as in, "If I had to explain how a P&L statement works to an AI engineer", training is R&D cost and inference cost is COGS.

Re: Are OpenAI and Anthropic losing money on inference?

#238

Post by otterley » Thu, Aug 28, 2025, 4:56 PM UTC

tsunamifury wrote: post #226

otterley wrote: post #124

Earlier quoted context omitted.

> As someone who has been taking the largest part of Google and facebooks ad wallet share away Can you elaborate? You’ve sparked my curiosity.

There are two companies gaining significant wallet share: Amazon and TikTok. Of those only one is taking a significant early share of both Google and Facebook.

OK, but you are a person, not a company. "You" are not taking the share away.

Re: Are OpenAI and Anthropic losing money on inference?

#239

Post by agf » Thu, Aug 28, 2025, 4:58 PM UTC

Onavo wrote: post #227

qrios wrote: post #203

For sure an interesting calculation. Only one remark from someone with GPU metal experience: > But compute becomes the bottleneck in certain scenarios. With long context sequences, attention computation scales quadratically with sequence length. Even if the statement about quadratically scales is right, the bottleneck we are talking about is somewhere north by factor 1000. If 10k cores do only simple matrix operation…

What do you mean by collision?

I believe it's that the bus can only serve one chip at a time, so it has to actually be faster since sometimes one chip's data will have to wait for the data of another chip to finish first.

Re: Are OpenAI and Anthropic losing money on inference?

#240

Post by jgalt212 » Thu, Aug 28, 2025, 4:59 PM UTC

gomox wrote: post #237

drob518 wrote: post #223

Earlier quoted context omitted.

Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and therefore basically meaningless.

I can't imagine the hoops an accountant would have to go through to argue training cost is COGS. In the most obvious stick-figures-for-beginners interpretation, as in, "If I had to explain how a P&L statement works to an AI engineer", training is R&D cost and inference cost is COGS.

there's not a bright line there, though.