From https://www.theverge.com/command-line-newsletter/759897/sam-..., Sam Altman said:
> “If we didn’t pay for training, we’d be a very profitable company.”
Are OpenAI and Anthropic losing money on inference?
21–30 of 495 posts
Re: Are OpenAI and Anthropic losing money on inference?
#22These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…
> You have to include all the costs associated with the model, not just inference. The title of the article directly says “on inference”. It’s not a mistake to exclude training costs. This is about incremental costs of inference.
Hacker News commenters just can't help but critique things even when they're missing the point
Re: Are OpenAI and Anthropic losing money on inference?
#23Earlier quoted context omitted.
I think the point isn't to argue AI companies are money printers or even that they're fairly valued, it's that at least the unit economics work out. Contrast this to something like moviepass, where they were actually losing money on each subscriber. Sure, a company that requires huge capital investments that might never be paid back isn't great either, but at least it's better than moviepass.
Unit economics needs to include the cost of the thing being sold, not just the direct cost of selling it. Unit economics is mostly a manufacturing concept and the only reason it looks OK here is because of not really factoring in the cost of building the thing into the cost of the thing. Someone might say I don’t understand “unit economics” but I’d simply argue applying a unit economics argument saying it’s good with…
You can amortise the training cost across billions of inference requests though. It's the marginal cost for inference that's most interesting here.
Re: Are OpenAI and Anthropic losing money on inference?
#24These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…
Their assumption is that training is a fixed cost: you'll spend the same amount on training for 5 users as you will with 500 million users.
Spending hundreds of millions of dollars on training when you are two guys in a garage is quite significant, but the same amount is absolutely trivial if you are planet-scale.
The big question is: how will training cost develop? Best-case scenario is a one-and-done run. But we're now seeing an arms race between the various AI providers: worst-case scenario, can the market survive an exponential increase in training costs for sublinear improvements?
Re: Are OpenAI and Anthropic losing money on inference?
#25I don't believe the asymmetry between prefill and decode is that large. If it were, it would make no sense for most of the providers to have separate pricing for prefill with cache hits vs. without. Given the analysis is based on R1, Deepseek's actual in-production numbers seem highly relevant: https://github.com/deepseek-ai/open-infra-index/blob/main/20... (But yes, they claim 80% margins on the compute in that arti…
Yes. I was really surprised at this myself (author here). If you have some better numbers I'm all ears. Even on my lowly 9070XT I get 20x the tok/s input vs output, and I'm not doing batching or anything locally.
I think the cache hit vs miss stuff makes sense at >100k tokens where you start getting compute bound.
Re: Are OpenAI and Anthropic losing money on inference?
#26Earlier quoted context omitted.
Unit economics needs to include the cost of the thing being sold, not just the direct cost of selling it. Unit economics is mostly a manufacturing concept and the only reason it looks OK here is because of not really factoring in the cost of building the thing into the cost of the thing. Someone might say I don’t understand “unit economics” but I’d simply argue applying a unit economics argument saying it’s good with…
But what about running Deepseek R1 or (insert other open weights model here)? There is no training cost for that.
1. Someone is still paying for that cost.
2. “Open source” is great but then it’s just a commodity. It would be very hard to build a sustainable business purely on the back of commoditized models. Adding a feature to an actual product that does something else though? Sure.
Re: Are OpenAI and Anthropic losing money on inference?
#27If inference is that cheap, why is not even one company profitable yet?
Because they're spending it all on training the next model.
Re: Are OpenAI and Anthropic losing money on inference?
#28Earlier quoted context omitted.
I think the point isn't to argue AI companies are money printers or even that they're fairly valued, it's that at least the unit economics work out. Contrast this to something like moviepass, where they were actually losing money on each subscriber. Sure, a company that requires huge capital investments that might never be paid back isn't great either, but at least it's better than moviepass.
Unit economics needs to include the cost of the thing being sold, not just the direct cost of selling it. Unit economics is mostly a manufacturing concept and the only reason it looks OK here is because of not really factoring in the cost of building the thing into the cost of the thing. Someone might say I don’t understand “unit economics” but I’d simply argue applying a unit economics argument saying it’s good with…
The cost of “manufacturing” an AI response is the inference cost, which this article covers.
> That would be like saying the unit economics of selling software is good because the only cost is some bandwidth and credit card processing fees. You need to include the cost of making the software
Unit economics is about the incremental value and costs of each additional customer.
You do not amortize the cost of software into the unit economics calculations. You only include the incremental costs of additional customers.
> just like you need to include the cost of making the models.
The cost of making the models is important overall, but it’s not included in the unit economics or when calculating the cost of inference.
Re: Are OpenAI and Anthropic losing money on inference?
#29These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…
What will be the knock on effect on us consumers?
Self hosting LLMs isn’t completely out of the realm of feasibility. Hardware cost may be 2-3x a hardcore gaming rig but it would be neat to see open source, self hosted, coding helpers. When Linux hit the scenes it put UNIX(ish) power in the hands of anyone with no license fee required. Surely somewhere someone is doing the same with LLM assisted coding.
Re: Are OpenAI and Anthropic losing money on inference?
#30Earlier quoted context omitted.
> You have to include all the costs associated with the model, not just inference. The title of the article directly says “on inference”. It’s not a mistake to exclude training costs. This is about incremental costs of inference.
Hacker News commenters just can't help but critique things even when they're missing the point
The parent commenter’s responses are all based on a wrong understanding of what unit economics means.
You don’t include fixed costs in the unit economics. Unit economics is about incremental costs.