Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

71–80 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#71

Post by sailingparrot »

gpjanik wrote: post #42

"Here's the key insight: each forward pass processes ALL tokens in ALL sequences simultaneously." This sounds incorrect, you only process all tokens once, and later incrementally. It's an auto-regressive model after all.

Not during prefill, i.e. the very first token generated in a new conversation. During this forward pass, all tokens in the context are all processed at the same time, and then attention's KV are cached, you still generate a single token, but you need to compute attention from all tokens to all tokens.

From that point on every subsequent tokens is processed sequentially in autoregressive way, but because we have the KV cache, this becomes O(N) (1 token query to all tokens) and not O(N^2)

Re: Are OpenAI and Anthropic losing money on inference?

#72

Post by wongarsu »

esafak wrote: post #41
Aurornis wrote: post #37

Earlier quoted context omitted.

> The models as is are still hugely useful, even if no further training was done. Exactly. The parent comment has an incorrect understanding of what unit economics means. The cost of training is not a factor in the marginal cost of each inference or each new customer. It’s unfortunate this comment thread is the highest upvoted right now when it’s based on a basic misunderstanding of unit economics.

The marginal cost is not the salient factor when the model has to be frequently retrained at great cost. Even if the marginal cost was driven to zero, would they profit?

But they don't have to be retained frequently at great cost. Right now they are retrained frequently because everyone is frequently coming out with new models and nobody wants to fall behind. But if investment for AI were to dry up everyone would stop throwing so much money at R&D, and if everyone else isn't investing in new models you don't have to either. The models are powerful as they are, most of the knowledge in them isn't going to rapidly obsolete, and where that is a concern you can paper over it with RAG or MCP servers. If everyone runs out of money for R&D at the same time we could easily cut back to a situation where we get an updated version of the same model every 3 years instead of a bigger/better model twice a year.

And whether companies can survive in that scenario depends almost entirely on their unit economics of inference, ignoring current R&D costs

Re: Are OpenAI and Anthropic losing money on inference?

#73

Post by nixgeek »

lkjdsklf wrote: post #48
JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

It’s fun to work backwards, but i was listening to a podcast where the journalists were talking about a dinner that Sam Altman had. This question came up and Sam said they were profitable if you exclude training and the COO corrected him So at least for OpenAI, the answer is “no” They did say it was close And that’s if you exclude training costs which is kind of absurd because it’s not like you can stop training

Excluding training two of their biggest costs will be payroll and inferencing for all the free users.

It’s therefore interesting that they claimed it was close: this supports the theory inferencing from paid users is a (big) money maker if it’s close to covering all the free usage and their payroll costs?

Re: Are OpenAI and Anthropic losing money on inference?

#74

Post by AstroBen »

gitremote wrote: post #45

These numbers are off. > $20/month ChatGPT Pro user: Heavy daily usage but token-limited ChatGPT Pro is $200/month and Sam Altman already admitted that OpenAI is losing money from Pro subscriptions in January 2025: "insane thing: we are currently losing money on openai pro subscriptions! people use it much more than we expected." - Sam Altman, January 6, 2025 https://xcancel.com/sama/status/1876104315296968813

I just straight up don't trust him

Saying that is the equivalent of him saying "our product is really valuable! use it!"

Re: Are OpenAI and Anthropic losing money on inference?

#75

Post by furyofantares »

JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

I don't think it's an accounting error when the article title says "Are OpenAI and Anthropic Really Losing Money on Inference?"

And it's a relevant question because people constantly say these companies are losing money on inference.

Re: Are OpenAI and Anthropic losing money on inference?

#76

Post by scrollaway »

martinald wrote: post #13
JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

(Author here). Yes I am aware of that and did mention it. However - what I wanted to push back in this article was that claude code was completely unsustainable and therefore a flash in the pan and devs aren't at risk (I know you are not saying this). The models as is are still hugely useful, even if no further training was done.

> claude code was completely unsustainable and therefore a flash in the pan and devs aren't at risk

How can you possibly say this if you know anything about the evolution of costs in the past year?

Inference costs are going down constantly, and as models get better they make less mistakes which means less cycles = less inference to actually subsidize.

This is without even looking at potential fundamental improvements in LLMs and AI in general. And with all the trillions in funding going into this sector, you can't possibly think we're anywhere near the technological peak.

Speaking as a founder managing multiple companies: Claude Code's value is in the thousands per month /per person/ (with the proper training). This isn't a flash in the pan, this isn't even a "prediction" - the game HAS changed and anyone telling you it hasn't is trying to cover their head with highly volatile sand.

Re: Are OpenAI and Anthropic losing money on inference?

#77

Post by nixgeek »

JCM9 wrote: post #15
politelemon wrote: post #6

Earlier quoted context omitted.

What will be the knock on effect on us consumers?

Costs will go up to levels where people will no longer find this stuff as useful/interesting. It’s all fun and games until the subsides end. See the recent reactions to AWS pricing on Kiro where folks had a big WTF reaction on pricing after, it appears, AWS tried to charge realistic pricing based on what this stuff actually costs.

Isn’t AWS always quite expensive? Look at their margins and the amount of cash it throws off, versus the consumer/retail business which runs a ton more revenue but no profit.

If you’re applying the same pricing structure to Kiro as to all AWS products then, yeah, it’s not particularly hobbyist accessible?

Re: Are OpenAI and Anthropic losing money on inference?

#78

Post by benreesman »

JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

My observation is that Opus is chronically capacity constrained while being dramatically more expensive than any of the others.

To me that more or less settles both "which one is best" and "is it subsidized".

Can't be sure, but anything else defies economic gravity.

Re: Are OpenAI and Anthropic losing money on inference?

#79

Post by techpineapple »

Aurornis wrote: post #64
pityJuke wrote: post #21

From https://www.theverge.com/command-line-newsletter/759897/sam-... , Sam Altman said: > “If we didn’t pay for training, we’d be a very profitable company.”

Exactly. All of the claims that OpenAI is losing money on every request are wrong. OpenAI hasn’t even unlocked all of their possible revenue opportunities from the free tier such as ads (like Google search), affiliate links, and other services. There’s also a lot of comments in this thread who want LLM companies to fail for different reasons, so they’re projecting that wish on to imagined unit economics. I’m having f…

Because Sam Altman said so?

Sam Altman also said this:

https://xcancel.com/sama/status/1876104315296968813

Re: Are OpenAI and Anthropic losing money on inference?

#80

Post by grim_io »

Workaccount2 wrote: post #52
chasd00 wrote: post #29

Earlier quoted context omitted.

Self hosting LLMs isn’t completely out of the realm of feasibility. Hardware cost may be 2-3x a hardcore gaming rig but it would be neat to see open source, self hosted, coding helpers. When Linux hit the scenes it put UNIX(ish) power in the hands of anyone with no license fee required. Surely somewhere someone is doing the same with LLM assisted coding.

The only reason to have a local model right now is for privacy and hobby. The economics are awful and local model performance is pretty lackluster by comparison. Never mind much slower and narrower context length. $6,000 is 2.5 years of a $200/mo subscription. And in 2.5 years that $6k setup will likely be equivalent to a $1k setup of the time.

We don't even need to compare it to the most expensive subscriptions.

The $20 subscription is far more capable than anything i could build locally for under $10k.

Post reply on HN