Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

51–60 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#51

Post by Aurornis »

esafak wrote: post #41
Aurornis wrote: post #37

Earlier quoted context omitted.

> The models as is are still hugely useful, even if no further training was done. Exactly. The parent comment has an incorrect understanding of what unit economics means. The cost of training is not a factor in the marginal cost of each inference or each new customer. It’s unfortunate this comment thread is the highest upvoted right now when it’s based on a basic misunderstanding of unit economics.

The marginal cost is not the salient factor when the model has to be frequently retrained at great cost. Even if the marginal cost was driven to zero, would they profit?

Unit economics are the salient factor of inference costs, which this article is about.

Re: Are OpenAI and Anthropic losing money on inference?

#52

Post by Workaccount2 »

chasd00 wrote: post #29
politelemon wrote: post #6

Earlier quoted context omitted.

What will be the knock on effect on us consumers?

Self hosting LLMs isn’t completely out of the realm of feasibility. Hardware cost may be 2-3x a hardcore gaming rig but it would be neat to see open source, self hosted, coding helpers. When Linux hit the scenes it put UNIX(ish) power in the hands of anyone with no license fee required. Surely somewhere someone is doing the same with LLM assisted coding.

The only reason to have a local model right now is for privacy and hobby.

The economics are awful and local model performance is pretty lackluster by comparison. Never mind much slower and narrower context length.

$6,000 is 2.5 years of a $200/mo subscription. And in 2.5 years that $6k setup will likely be equivalent to a $1k setup of the time.

Re: Are OpenAI and Anthropic losing money on inference?

#53

Post by Workaccount2 »

jonathan-adly wrote: post #16

Basically- the same math as modern automated manufacturing. Super expensive and complex build-out - then a money printer once running and optimized. I know there is lots of bearish sentiments here. Lots of people correctly point out that this is not the same math as FAANG products - then they make the jump that it must be bad. But - my guess is these companies end up with margins better than Tesla (modern manufacture…

The difference is the money printer right now only prints for ~6 months before it needs to be replaced with an even more expensive printer.

Re: Are OpenAI and Anthropic losing money on inference?

#54

Post by [deleted] »

Aurornis wrote: post #51
esafak wrote: post #41

Earlier quoted context omitted.

The marginal cost is not the salient factor when the model has to be frequently retrained at great cost. Even if the marginal cost was driven to zero, would they profit?

Unit economics are the salient factor of inference costs, which this article is about.

[deleted]

Re: Are OpenAI and Anthropic losing money on inference?

#55

Post by artursapek »

Aurornis wrote: post #30
artursapek wrote: post #22

Earlier quoted context omitted.

Hacker News commenters just can't help but critique things even when they're missing the point

The parent commenter’s responses are all based on a wrong understanding of what unit economics means. You don’t include fixed costs in the unit economics. Unit economics is about incremental costs.

I know I'm agreeing with you. I'm saying, don't bother with him lol

Re: Are OpenAI and Anthropic losing money on inference?

#56

Post by rossdavidh »

"Heavy readers - applications that consume massive amounts of context but generate minimal output - operate in an almost free tier for compute costs."

Not saying there's not interesting analysis here, but this is assuming that they don't have to pay for access to the massive amounts of context. Sources like stackoverflow and reddit that used to be free, are not going to be available to keep the model up to date.

If this analysis is meant to say "they're not going to turn the lights out because of the costs of running", that may be so, but if they cannot afford to keep training new models every so often they will become less relevant over timte, and I don't know if they will get an ocean of VC money to do it all again (at higher cost than last time, because the sources want their cut now).

Re: Are OpenAI and Anthropic losing money on inference?

#57

Post by kgwgk »

Aurornis wrote: post #38
kgwgk wrote: post #35

Earlier quoted context omitted.

Your comment may apply to the original commenter “missing” the point of TFA and to the person replying “missing” the point of that comment. And to my comment “missing” the point of yours - which may have also “missed” the point.

I’ve clearly “missed” the point you were trying to make, because there’s nothing complicated: The article is about unit economics and marginal costs of inferences and this comment thread is trying to criticize the article based on a misunderstanding of what unit economics means.

I was not trying to make any point. I’m not even sure if the comment I replied to was suggesting that it was you or the other commenter who was missing some point or another.

Re: Are OpenAI and Anthropic losing money on inference?

#58

Post by jsnell »

JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

For the top few providers, the training is getting amortized over absurd amount of inference. E.g. Google recently mentioned that they processed 980T tokens over all surfaces in June 2025.

The leaked OpenAI financial projections for 2024 showed about equal amount of money spent on training and inference.

Amortizing the training per-query really doesn't meaningfully change the unit economics.

> Fact remains when all costs are considered these companies are losing money and so long as the lifespan of a model is limited it’s going to stay ugly. Using that apartment building analogy it’s like having to knock down and rebuild the building every 6 months to stay relevant. That’s simply not a viable business model.

To the extent they're losing money, it's because they're giving free service with no monetizaton to a billion users. But since the unit costs are so low, monetizing those free users with ads will be very lucrative the moment they decide to do so.

Re: Are OpenAI and Anthropic losing money on inference?

#59

Post by rich_sasha »

JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

> Fact remains when all costs are considered these companies are losing money

You would need to figure out what exactly they are losing money on. Making money on inference is like operating profit - revenue less marginal costs. So the article is trying to answer if this operating profit is positive or negative. Not whether they are profitable as a whole.

If things like cost of maintaining data centres or electricity or bandwidth push them into the red, then yes, they are losing money on inference.

If the things that make them lose money is new R&D then that's different. You could split them up into a profitable inference company and a loss making startup. Except the startup isn't purely financed by VC etc, but also by a profitable inference company.

Re: Are OpenAI and Anthropic losing money on inference?

#60

Post by jsnell »

martinald wrote: post #25
jsnell wrote: post #9

I don't believe the asymmetry between prefill and decode is that large. If it were, it would make no sense for most of the providers to have separate pricing for prefill with cache hits vs. without. Given the analysis is based on R1, Deepseek's actual in-production numbers seem highly relevant: https://github.com/deepseek-ai/open-infra-index/blob/main/20... (But yes, they claim 80% margins on the compute in that arti…

Yes. I was really surprised at this myself (author here). If you have some better numbers I'm all ears. Even on my lowly 9070XT I get 20x the tok/s input vs output, and I'm not doing batching or anything locally. I think the cache hit vs miss stuff makes sense at >100k tokens where you start getting compute bound.

I linked to the writeup by Deepseek with their actual numbers from production, and you want "better numbers" than that?!

> Each H800 node delivers an average throughput of ~73.7k tokens/s input (including cache hits) during prefilling or ~14.8k tokens/s output during decoding.

That's a 5x difference, not 1000x. It also lines up with their pricing, as one would expect.

(The decode throughputs they give are roughly equal to yours, but you're claiming a prefill performance 200x times higher than they can achieve.)

Post reply on HN