Are OpenAI and Anthropic losing money on inference? • Hacker News Board

Re: Are OpenAI and Anthropic losing money on inference?

#381

Post by _sword » Thu, Aug 28, 2025, 8:59 PM UTC

BlindEyeHalo wrote: post #303

_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

I spoke with management at a couple companies that were training models, and some of them expensed the model training in-period as R&D. That's why

Re: Are OpenAI and Anthropic losing money on inference?

#382

Post by _sword » Thu, Aug 28, 2025, 9:02 PM UTC

ozgune wrote: post #337

_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

I agree that you could get to high margins, but I think the modeling holds only if you're an AI lab operating at scale with a setup tuned for your model(s). I think the most open study on this one is from the DeepSeek team: https://github.com/deepseek-ai/open-infra-index/blob/main/20... For others, I think the picture is different. When we ran benchmarks on DeepSeek-R1 on 8x H200 SXM using vLLM, we got up to 12K tota…

I was modeling configurations purpose-built for running specific models in specific workloads. I was trying to figure out how much of a gross margin drag some software companies could have if they hosted their own models and served them up as APIs or as integrated copilots with their other offerings

Re: Are OpenAI and Anthropic losing money on inference?

#383

Post by mgh95 » Thu, Aug 28, 2025, 9:05 PM UTC

MontyCarloHall wrote: post #314

BlindEyeHalo wrote: post #303

Earlier quoted context omitted.

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

As long as models continue on their current rapid improvement trajectory, retraining from scratch will be necessary to keep up with the competition. As you said, that's such a huge amount of continual CapEx that it's somewhat meaningless to consider AI companies' financial viability strictly in terms of inference costs, especially because more capable models will likely be much more expensive to train. But at some po…

> If the latter, assuming the cost of fine-tuning is a fraction of the cost of training from scratch, the low cost of inference does indeed make a bullish case for these companies.

On the other hand, this may also turn into cost effective methods such as model distillation and spot training of large companies (similarly to Deepseek). This would erode the comparative advantage of Anthropic and OpenAI, and result in a pure value-add play for integration with data sources and features such as SSO.

It isn't clear to me that a slowing of retraining will result in advantages to incumbents if model quality cannot be readily distinguished by end-users.

Re: Are OpenAI and Anthropic losing money on inference?

#384

Post by yalogin » Thu, Aug 28, 2025, 9:06 PM UTC

Romario77 wrote: post #377

yalogin wrote: post #373

Will these companies ever stop training new models? What does it mean if we get there. Feels like they will have to constantly train and improve the models, not sure what that means either. What ncremental improvements can these models show? Another question is - will it ever become less costly to train? Let to see opinions from someone in the know

current way the models works is that they don't have memory, it's included in training (or has to be provided as context). So to keep up with times the models have to be constantly trained. One thing though is that right now it's not just incremental training, the whole thing gets updated - multiple parameters and how the model is trained is different. This might not be the case in the future where the training could…

Sure the training can be made efficient, but how much better can these LLMs get in functionality?

Re: Are OpenAI and Anthropic losing money on inference?

#385

Post by Romario77 » Thu, Aug 28, 2025, 9:06 PM UTC

runako wrote: post #374

renjimen wrote: post #289

Earlier quoted context omitted.

But new models to date have cost more than the previous ones to create, often by an order of magnitude, so the shoe metaphor falls apart. A better metaphor would be oil and gas production, where existing oil and gas fields are either already finished (i.e. model is no longer SOTA -- no longer making a return on investment) or currently producing (SOTA inference -- making a return on investment). The key similarity wi…

> new models to date have cost more than the previous ones to create This largely was the case in software in the '80s-'10s (when versions largely disappeared) and still is the case in hardware. iPhone 17 will certainly cost far more to develop than did iPhone 10 or 5. iPhone 5 cost far more than 3G, etc.

I don't think it's the case if you take inflation into account.

You could see here: https://www.reddit.com/r/dataisbeautiful/comments/16dr1kb/oc...

new ones are generally cheaper if adjusted for inflation. This is a sale price, but assuming that margins stay the same it should reflect the manufacturing price. And from what I remember about apple earnings their margins increased over time, so it means the new phones are even cheaper. Which kind of makes sense.

Re: Are OpenAI and Anthropic losing money on inference?

#386

Post by kgwgk » Thu, Aug 28, 2025, 9:23 PM UTC

dcre wrote: post #356

kgwgk wrote: post #339

Earlier quoted context omitted.

>> If we didn't pay for training, we'd be a very profitable company. > ICYMI, Amodei said the same No. He says that even paying for training a model is profitable. It makes more revenue that it costs - all things considered. A much stronger claim.

I take them to be saying the same thing — the difference is that Altman is referring to the training of the next model happening now, while Amodei is referring to the training months ago of the model you're currently earning money back on through inference.

Maybe he means that but the quote says “We're profitable on inference.” - not “We're profitable on inference including training of that model.”

Re: Are OpenAI and Anthropic losing money on inference?

#387

Post by tick_tock_tick » Thu, Aug 28, 2025, 9:30 PM UTC

lawlessone wrote: post #363

tick_tock_tick wrote: post #342

Earlier quoted context omitted.

Because the law as it stands says they aren't and Congress may make no ex post facto law.

ok, but that's just one country.

I mean for AI literally the only countries involved are the USA and China. I doubt you think China is going to start respecting IP rights anytime soon.

Re: Are OpenAI and Anthropic losing money on inference?

#388

Post by hdgvhicv » Thu, Aug 28, 2025, 9:31 PM UTC

jedberg wrote: post #205

noodletheworld wrote: post #145

Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap,…

https://lambda.chat Deepseek R1 for free.

> I'm here to provide helpful, respectful, and appropriate content for all users. If you have any other requests or need assistance with a different type of story or topic, feel free to ask!

Re: Are OpenAI and Anthropic losing money on inference?

#389

Post by 9cb14c1ec0 » Thu, Aug 28, 2025, 9:37 PM UTC

atleastoptimal wrote: post #365

Everyone claiming AI companies are a financial ticking time bomb are using the same logic people used back in the 2000s when they claimed Amazon “never made a profit” and thus was a bad investment.

Wrong. The depreciation cost on the hundreds of billions of dollars spent on the AI build-out is almost certainly larger then the AI industry's gross income. This is a vastly different depreciation cost schedule than AWS.

Re: Are OpenAI and Anthropic losing money on inference?

#390

Post by runako » Thu, Aug 28, 2025, 9:38 PM UTC

Romario77 wrote: post #385

runako wrote: post #374

Earlier quoted context omitted.

> new models to date have cost more than the previous ones to create This largely was the case in software in the '80s-'10s (when versions largely disappeared) and still is the case in hardware. iPhone 17 will certainly cost far more to develop than did iPhone 10 or 5. iPhone 5 cost far more than 3G, etc.

I don't think it's the case if you take inflation into account. You could see here: https://www.reddit.com/r/dataisbeautiful/comments/16dr1kb/oc... new ones are generally cheaper if adjusted for inflation. This is a sale price, but assuming that margins stay the same it should reflect the manufacturing price. And from what I remember about apple earnings their margins increased over time, so it means the new phones a…

I should have addressed this. This thread is about the capital costs of getting to the first sale, so that's model training for an LLM vs all the R&D in an iPhone.

Recent iPhones use Apple's own custom silicon for a number of components, and are generally vastly more complex. The estimates I have seen for iPhone 1 development range from $150 million to $2.5 billion. Even adjusting for inflation, a current iPhone generation costs more than the older versions.

And it absolutely makes sense for Apple to spend more in total to develop successive generations, because they have less overall product risk and larger scale to recoup.