Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

311–320 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#311

Post by true_religion »

meshugaas wrote: post #262
dcre wrote: post #219

Earlier quoted context omitted.

ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…

The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…

It's model as a company because people are using the VC mentality, and also explaining competition.

Model as a product is the reality, but each model competes with previous models and is only successful if it's both more cost effective, and also more effective in general at its tasks. By the time you get to model Z, you'll never use model A for any task as the model lineage cannibalizes sales of itself.

Re: Are OpenAI and Anthropic losing money on inference?

#312

Post by overgard »

snowwrestler wrote: post #275

Not wishing to do a shallow dismissal here, but I always assumed AI must be profitable on inference otherwise no one would pursue it as a business given how expensive the training is. It seems sort of like wondering if a fiber ISP is profitable per GB bandwidth. Of course it is; the expensive part is getting the fiber to all the homes. So the operations must be profitable or there is simply no business model possible…

AI right now seems more like a religious movement than a business one. It doesn't matter how much it costs (to the true believers), its about getting to AGI first.

Re: Are OpenAI and Anthropic losing money on inference?

#313

Post by DenisM »

drob518 wrote: post #288
gomox wrote: post #237

Earlier quoted context omitted.

I can't imagine the hoops an accountant would have to go through to argue training cost is COGS. In the most obvious stick-figures-for-beginners interpretation, as in, "If I had to explain how a P&L statement works to an AI engineer", training is R&D cost and inference cost is COGS.

I wasn’t using COGS in a GAAP sense, but rather as a synonym for unspecified “costs.” My bad. I suppose you would classify training as development and ongoing datacenter and GPU costs as actual GAAP COGS. My point was, if all you focus on is revenue and ignore the costs of creating your business and keeping it running, it’s pretty easy for any business to be “profitable.”

It’s generally useful to consider unit economy separate from whole company. If your unit economy is negative thing are very bleak. If it’s positive, your chance are going up by a lot - scaling the business amortizes fixed (non-unit) costs, such as admin and R&D, and slightly improves unit margins as well.

However this does not work as well if your fixed (non-unit) cost is growing exponentially. You can’t get out of this unless your user base grows exponentially or the customer value (and price) per user grows exponentially.

I think this is what Altman is saying - this is an unusual situation: unit economy is positive but fixed costs are exploding faster than economy if scale can absorb it.

You can say it’s splitting hair, but insightful perspective often requires teasing things apart.

Re: Are OpenAI and Anthropic losing money on inference?

#314

Post by MontyCarloHall »

BlindEyeHalo wrote: post #303
_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

As long as models continue on their current rapid improvement trajectory, retraining from scratch will be necessary to keep up with the competition. As you said, that's such a huge amount of continual CapEx that it's somewhat meaningless to consider AI companies' financial viability strictly in terms of inference costs, especially because more capable models will likely be much more expensive to train.

But at some point, model improvement will saturate (perhaps it already has). At that point, model architecture could be frozen, and the only purpose of additional training would be to bake new knowledge into existing models. It's unclear if this would require retraining the model from scratch, or simply fine-tuning existing pre-trained weights on a new training corpus. If the former, AI companies are dead in the water, barring a breakthrough in dramatically reducing training costs. If the latter, assuming the cost of fine-tuning is a fraction of the cost of training from scratch, the low cost of inference does indeed make a bullish case for these companies.

Re: Are OpenAI and Anthropic losing money on inference?

#315

Post by reenorap »

An interesting exercise would be what prompts would create the most costs for LLMs but outwards little to no costs for the issuers. Are all prompts equal and the only factor being the lengths of the input and output prompts? Or is there processing of the prompts that could be exceedingly expensive for the LLM?

Re: Are OpenAI and Anthropic losing money on inference?

#316

Post by DenisM »

ToucanLoucan wrote: post #235
dcre wrote: post #229

Earlier quoted context omitted.

The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.

So is OpenAI capable of not making a new model at some point? They've been training the next model continuously as long as they've existed AFAIK. Our software house spends a lot on R&D sure, but we're still incredibly profitable all the same. If OpenAI is in a position where they effectively have to stop iterating the product to be profitable, I wouldn't call that a very good place to be when you're on the verge of h…

There’s still untapped value in deeper integrations. They might hit a jackpot of exponentially increasing value from network effects caused by tight integration with e.g. disjoint business processes.

We know that businesses with tight network effects can grow to about 2 trillion in valuation.

Re: Are OpenAI and Anthropic losing money on inference?

#317

Post by trilogic »

_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

I have to disagree. The biggest cost is still energy consumption, water and maintenance. Not to mention, to keep up with the rivals in incredibly high tempo (so offering billions like Meta recently). Then the cost of hardware that is equal to Nvidia skyrocketing shares :) No one should dare to talk about profit yet. Now is time to grab the market, invest a lot and work hard, hopping for a future profit. The equation is still work on progress.

Re: Are OpenAI and Anthropic losing money on inference?

#318

Post by wtallis »

trilogic wrote: post #317
_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

I have to disagree. The biggest cost is still energy consumption, water and maintenance. Not to mention, to keep up with the rivals in incredibly high tempo (so offering billions like Meta recently). Then the cost of hardware that is equal to Nvidia skyrocketing shares :) No one should dare to talk about profit yet. Now is time to grab the market, invest a lot and work hard, hopping for a future profit. The equation…

> The biggest cost is still energy consumption, water and maintenance.

Are you saying that the operating costs for inference exceed the costs of training?

Re: Are OpenAI and Anthropic losing money on inference?

#319

Post by colinsane »

noodletheworld wrote: post #145

Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap,…

> I should be able to get a cheap / run my own 600B param model.

if the margins on hosted inference are 80%, then you need > 20% utilization of whatever you build for yourself for this to be less costly to you (on margin).

i self-host open weight models (please: deepseek et al aren't open _source_) on whatever $300 GPU i bought a few years ago, but if it outputs 2 tokens/sec then i'm waiting 10 minutes for most results. if i want results in 10s instead of 10m, i'll be paying $30000 instead. if i'm prompting it 100 times during the day, then it's idle 99% of the time.

coordinating a group buy for that $30000 GPU and sharing that across 100 people probably makes more sense than either arrangement in the previous paragraph. for now, that's a big component of what model providers, uh, provide.

Re: Are OpenAI and Anthropic losing money on inference?

#320

Post by tsunamifury »

otterley wrote: post #238
tsunamifury wrote: post #226

Earlier quoted context omitted.

There are two companies gaining significant wallet share: Amazon and TikTok. Of those only one is taking a significant early share of both Google and Facebook.

OK, but you are a person, not a company. "You" are not taking the share away.

"I'm digging a trench"

"No you're not, WE are digging a trench!"

Yes fine, but "I am as well".

Sheesh. Also I, personally, do and lead the work of taking the wallet share. So I will stick with "I" and would accept any of my team saying the same.

Post reply on HN