Are OpenAI and Anthropic losing money on inference? • Hacker News Board

Re: Are OpenAI and Anthropic losing money on inference?

#201

Post by thatguysaguy » Thu, Aug 28, 2025, 4:10 PM UTC

sc68cal wrote: post #125

This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.

Why would you think that deepseek is more efficient than gpt-5/Claude 4 though? There's been enough time to integrate the lessons from deepseek.

Re: Are OpenAI and Anthropic losing money on inference?

#202

Post by hirako2000 » Thu, Aug 28, 2025, 4:11 PM UTC

losvedir wrote: post #47

JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

I think this is missing the point that the very interesting article makes. You're arguing that maybe the big companies won't recoup their investment in the models, or profitably train new ones. But that's a separate question. Whether a model - which now exists! - can profitably be run is very good to know. The fact that people happily pay more than the inference costs means what we have now is sustainable. Maybe Anth…

It hasn't even proven that, it's assuming a ridiculous daily usage, and also ignoring free riders. Running a model is likely not profitable for any provider right now. Even a public company (e.g alphabet) isn't obliged to honest figures since numbers on the sheets can be moved left and right. We won't know for a other year or two when companies we have today start falling and their founders start talking.

Re: Are OpenAI and Anthropic losing money on inference?

#203

Post by qrios » Thu, Aug 28, 2025, 4:13 PM UTC

For sure an interesting calculation. Only one remark from someone with GPU metal experience:

> But compute becomes the bottleneck in certain scenarios. With long context sequences, attention computation scales quadratically with sequence length.

Even if the statement about quadratically scales is right, the bottleneck we are talking about is somewhere north by factor 1000. If 10k cores do only simple matrix operations each needs to have new data (up to 64k) available every 500 cycles (let's say). Getting these amount of data (without _any_ collision) means something like 100+GByte/s per core. Even 2+TByte/s on HBM means the bottleneck is the memory transfer rate, by something like 500 times. With collision, we talk about an additional factor like 5000 (last time I've done some tests with a 4090).

Re: Are OpenAI and Anthropic losing money on inference?

#204

Post by eikenberry » Thu, Aug 28, 2025, 4:13 PM UTC

Aurornis wrote: post #64

pityJuke wrote: post #21

From https://www.theverge.com/command-line-newsletter/759897/sam-... , Sam Altman said: > “If we didn’t pay for training, we’d be a very profitable company.”

Exactly. All of the claims that OpenAI is losing money on every request are wrong. OpenAI hasn’t even unlocked all of their possible revenue opportunities from the free tier such as ads (like Google search), affiliate links, and other services. There’s also a lot of comments in this thread who want LLM companies to fail for different reasons, so they’re projecting that wish on to imagined unit economics. I’m having f…

So inference is cheap but training is expensive and getting more expensive. It seems like if they can't get training expenses down, cheap inference won't matter.

Re: Are OpenAI and Anthropic losing money on inference?

#205

Post by jedberg » Thu, Aug 28, 2025, 4:13 PM UTC

noodletheworld wrote: post #145

Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap,…

https://lambda.chat

Deepseek R1 for free.

Re: Are OpenAI and Anthropic losing money on inference?

#206

Post by mrcwinn » Thu, Aug 28, 2025, 4:13 PM UTC

As the author seems to admit, an outsider is going to lack so much information (costs, loss leaders, etc), one has to assume any modeling is so inaccurate that it's not worth anything.

So the question remains unanswered, at least for us. For those putting money in, you can be absolutely certain they have a model with sufficient data to answer the question. Since money did go in, even if it's venture, the answer is probably "yes in the immediate, but no over time."

Re: Are OpenAI and Anthropic losing money on inference?

#207

Post by basilgohar » Thu, Aug 28, 2025, 4:14 PM UTC

baxtr wrote: post #193

phillipcarter wrote: post #152

Earlier quoted context omitted.

Uhhh, I'm pretty sure DeepSeek shook the industry because of a 14x reduction in training cost, not inference cost. We also don't know the per-token cost for OpenAI and Anthropic models, but I would be highly surprised if it was significantly more expensive than open models anyone can use and run themselves. It's not like they're also not investing in inference research.

Because of the alleged reduction in training costs.

All reports by companies are alleged until verified by other, more trustworthy sources. I don't think it's especially notable that it's alleged because it's DeepSeek vs. the alleged numbers from other companies.

Re: Are OpenAI and Anthropic losing money on inference?

#208

Post by gmd63 » Thu, Aug 28, 2025, 4:15 PM UTC

phillipcarter wrote: post #152

sc68cal wrote: post #125

This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.

Uhhh, I'm pretty sure DeepSeek shook the industry because of a 14x reduction in training cost, not inference cost. We also don't know the per-token cost for OpenAI and Anthropic models, but I would be highly surprised if it was significantly more expensive than open models anyone can use and run themselves. It's not like they're also not investing in inference research.

DeepSeek was trained with distillation. Any accurate estimate of training costs should include the training costs of the model that it was distilling.

Re: Are OpenAI and Anthropic losing money on inference?

#209

Post by dragonwriter » Thu, Aug 28, 2025, 4:16 PM UTC

booi wrote: post #198

dragonwriter wrote: post #177

Earlier quoted context omitted.

> but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices There are multiple API providers offering models at dirt cheap prices, enough so that there is at least one well-known API provider that is an aggreggator of other API providers that offers lots of models at $0. > The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1?…

you can also run deepseek for free on a modestly sized laptop

At 4-bit quant, R1 takes 300+ gigs just for weights. You can certainly run smaller models into which R1 has been distilled on a modest laptop, but I don't see how you can run R1 itself on anything that wouldn't be considered extreme for a laptop in at least one dimension.

Re: Are OpenAI and Anthropic losing money on inference?

#210

Post by hirako2000 » Thu, Aug 28, 2025, 4:17 PM UTC

empath75 wrote: post #179

hirako2000 wrote: post #164

Earlier quoted context omitted.

Imo the article is totally off the mark since it assumes users on average do not go over th 1M tokens per day. Afaik openai doesn't enforce a daily quota even on the $20 plans unless the platform is under pressure. Since I often consume 20M token per day, one can assume many would use far more than the 1M tokens assumed in the article's calculations.

There's zero basis for assuming any of that. The most likely situation is a power law curve where the vast majority of users don't use it much at all and the top 10% of users account for 90% of the usage. It is very likely that you are in the top 10% of users.

True. the article also has zero basis in its estimating the average usage from each tier's user base.

I somewhat doubt my usage is so close to the edge of the curve since I don't even pay for any plan. It could be that I'm very frugal with money and fat on consumption while most are more balanced, but 1M token per day in any case sounds slim for any user who pays for the service.