Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

321–330 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#321

Post by vonneumannstan »

BlindEyeHalo wrote: post #303
_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

I suspect we've already reached the point with models at the GPT5 tier where the average person will no longer recognize improvements and this model can be slightly improved at slow intervals and indeed run for years. Meanwhile research grade models will still need to be trained at massive cost to improve performance on relatively short time scales.

Re: Are OpenAI and Anthropic losing money on inference?

#322

Post by christina97 »

BlindEyeHalo wrote: post #303
_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

In the same way that every other startup tries to sweep R&D costs under the rug and say “yeah but the marginal unit economics have 50% gross margins, we’ll be a great business soon”.

Re: Are OpenAI and Anthropic losing money on inference?

#323

Post by gmd63 »

ffsm8 wrote: post #221
gmd63 wrote: post #208

Earlier quoted context omitted.

DeepSeek was trained with distillation. Any accurate estimate of training costs should include the training costs of the model that it was distilling.

That makes the calculation nonsensical, because if you go there... you'd also have to include all energy used in producing the content the other model providers used. So now suddenly everyones devices on which they wrote comments on social media, pretty much all servers to have ever served a request to open AI/Google/anthropics bots etc pp Seriously, that claim was always completely disingenuous

I don't think it's that nonsensical to realize that in order to have AI, you need generations of artists, journalists, scientists, and librarians to produce materials to learn from.

And when you're using an actual AI model to "train" (copy), it's not even a shred of nonsense to realize the prior model is a core component of the training.

Re: Are OpenAI and Anthropic losing money on inference?

#324

Post by vonneumannstan »

meshugaas wrote: post #262
dcre wrote: post #219

Earlier quoted context omitted.

ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…

The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…

>Also, in Nike's case, as they grow they get better at making more shoes for cheaper.

This is clearly the case for models as well. Training and serving inference for GPT4 level models is probably > 100x cheaper than they used to be. Nike has been making Jordan 1's for 40+ years! OpenAI would be incredibly profitable if they could live off the profit from improved inference efficiency on a GPT4 level model!

Re: Are OpenAI and Anthropic losing money on inference?

#325

Post by skybrian »

meshugaas wrote: post #262
dcre wrote: post #219

Earlier quoted context omitted.

ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…

The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…

Analogies don't prove anything, but they're still useful for suggesting possibilities for thinking about a problem.

If you don't like "model as company," how about "model as making a movie?" Any given movie could be profitable or not. It's not necessarily the case that movie budgets always get bigger or that an increased budget is what you need to attract an audience.

Re: Are OpenAI and Anthropic losing money on inference?

#326

Post by achenet »

simianwords wrote: post #200
chairhairair wrote: post #138

Earlier quoted context omitted.

Anyone paying attention should have zero trust in what Sam Altman says.

What do you think his strategy is? He has to make money at some point. I don’t buy the logic that he will “scam” his investors and run away at some point.

He makes money by convincing people to buy OpenAI stock.

If OpenAI goes down tomorrow, he will be just fine. His incentive is to sell the stock, not actually build and run a profitable business.

Look at Adam Neumann as an example of how to lose billions of investor dollars and still walk out of the ensuing crash with over a billion.

https://en.wikipedia.org/wiki/Adam_Neumann

His strategy is to sell OpenAI stock like it was Bitcoin in 2020, and if for some reason the market decides that maybe a company that loses large amounts of cash isn't actually a good investment... he'll be fine, he's had plenty of time to turn some of his stock into money :)

Re: Are OpenAI and Anthropic losing money on inference?

#327

Post by umpalumpaaa »

wtallis wrote: post #318
trilogic wrote: post #317

Earlier quoted context omitted.

I have to disagree. The biggest cost is still energy consumption, water and maintenance. Not to mention, to keep up with the rivals in incredibly high tempo (so offering billions like Meta recently). Then the cost of hardware that is equal to Nvidia skyrocketing shares :) No one should dare to talk about profit yet. Now is time to grab the market, invest a lot and work hard, hopping for a future profit. The equation…

> The biggest cost is still energy consumption, water and maintenance. Are you saying that the operating costs for inference exceed the costs of training?

No. But training an LLM is certainly very very expensive and a gamble every time you do it. I think of it a bit like a pharmaceutical company doing vaccine research…

Re: Are OpenAI and Anthropic losing money on inference?

#328

Post by vitaflo »

CjHuber wrote: post #146
sc68cal wrote: post #125

This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.

The reason it shook the market at least was because of the claim that its training cost was 5 million.

Also the fact that it cost 10% of what other models cost. Pretty much still does.

Re: Are OpenAI and Anthropic losing money on inference?

#329

Post by skybrian »

gitremote wrote: post #45

These numbers are off. > $20/month ChatGPT Pro user: Heavy daily usage but token-limited ChatGPT Pro is $200/month and Sam Altman already admitted that OpenAI is losing money from Pro subscriptions in January 2025: "insane thing: we are currently losing money on openai pro subscriptions! people use it much more than we expected." - Sam Altman, January 6, 2025 https://xcancel.com/sama/status/1876104315296968813

That's interesting but it doesn't mean they're losing money on the $20/month users. The Pro plan selects for heavy-usage enthusiasts.

Re: Are OpenAI and Anthropic losing money on inference?

#330

Post by oblio »

DenisM wrote: post #309
dcre wrote: post #219

Earlier quoted context omitted.

ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…

Fantastic perspective. Basically each new company puts competitive pressure on the previous company, and together they compress margins. They are racing themselves to the bottom. I imagine they know this and bet on AGI primacy.

> I imagine they know this and bet on AGI primacy.

Just like Uber and Tesla are betting on self driving cars. I think it's been 10 years now ("any minute now").

Post reply on HN