Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

81–90 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#81

Post by ninetyninenine »

Aurornis wrote: post #37
martinald wrote: post #13

Earlier quoted context omitted.

(Author here). Yes I am aware of that and did mention it. However - what I wanted to push back in this article was that claude code was completely unsustainable and therefore a flash in the pan and devs aren't at risk (I know you are not saying this). The models as is are still hugely useful, even if no further training was done.

> The models as is are still hugely useful, even if no further training was done. Exactly. The parent comment has an incorrect understanding of what unit economics means. The cost of training is not a factor in the marginal cost of each inference or each new customer. It’s unfortunate this comment thread is the highest upvoted right now when it’s based on a basic misunderstanding of unit economics.

I upvoted it because it aligns most closely with my own perspective. I have a strong dislike for AI and everything associated with it, so my judgment is shaped by that bias. If a post sounds realistic or complex, I have no interest in examining its nuance. I am not concerned with practical reality and prefer to accept it without thinking, so I support ideas that match my personal viewpoint.

I don’t understand why people like you have to call this stuff out? Like most of HN thinks the way I do and that’s why the post was upvoted. Why be a contrarian? There’s really no point.

Re: Are OpenAI and Anthropic losing money on inference?

#82

Post by toddmorey »

rich_sasha wrote: post #59
JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

> Fact remains when all costs are considered these companies are losing money You would need to figure out what exactly they are losing money on. Making money on inference is like operating profit - revenue less marginal costs. So the article is trying to answer if this operating profit is positive or negative. Not whether they are profitable as a whole. If things like cost of maintaining data centres or electricity…

Yes that's right. The inference costs in isolation are interesting because that speaks to the unit economics of this business: R&D / model training aside, can the service itself be scaled to operate at a profit? Because that's the only hope of all the R&D eventually paying dividends.

One thing that makes me suspect inference costs are coming down is how chatty the models have become lately, often appending encouragement to a checklist like "You can check off each item as you complete them!" Maybe I'm wrong, but I feel if inference was killing them, the responses would become more terse rather than more verbose.

Re: Are OpenAI and Anthropic losing money on inference?

#83

Post by OtherShrezzing »

This is a great article, but it doesn't appear to model H100 downtime in the $2/hr costs. It assumes that OpenAI and Anthropic can match demand for inference to their supply of H100s perfectly, 24/7, in all regions. Maybe you could argue that the idle H100s are being used for model training - but that's different to the article's argument that inference is economically sustainable in isolation.

Re: Are OpenAI and Anthropic losing money on inference?

#84

Post by techpineapple »

EcommerceFlow wrote: post #63

I wouldn't be surprised if their profit/query is at a negative for all major Ai companies, but guess what? They have a service which understands a users question/needs 100x better than a traditional Google search does. Once they tap into that for PPC/paid ads, their profit/query should jump into the green. In fact, there's a decent chance a lot of these models will go 100% free once that PPC pipeline is implemented a…

> They have a service which understands a users question/needs 100x better than a traditional Google search does.

Source?

Re: Are OpenAI and Anthropic losing money on inference?

#85

Post by JimDabell »

lkjdsklf wrote: post #48
JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

It’s fun to work backwards, but i was listening to a podcast where the journalists were talking about a dinner that Sam Altman had. This question came up and Sam said they were profitable if you exclude training and the COO corrected him So at least for OpenAI, the answer is “no” They did say it was close And that’s if you exclude training costs which is kind of absurd because it’s not like you can stop training

There’s no mention of that in this article about it:

https://archive.is/wZslL

They quote him as saying inference is profitable and end it at that.

Are you saying that the COO corrected him at the dinner, or on the podcast? Which podcast was it?

Re: Are OpenAI and Anthropic losing money on inference?

#86

Post by JCM9 »

furyofantares wrote: post #75
JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

I don't think it's an accounting error when the article title says "Are OpenAI and Anthropic Really Losing Money on Inference?" And it's a relevant question because people constantly say these companies are losing money on inference.

I think the nuance here is what people consider the “cost” of “inference.” Purely on compute costs and not accounting for the cost of the model (which is where the article focuses) it’s not bad.

Re: Are OpenAI and Anthropic losing money on inference?

#87

Post by churchill »

Workaccount2 wrote: post #53
jonathan-adly wrote: post #16

Basically- the same math as modern automated manufacturing. Super expensive and complex build-out - then a money printer once running and optimized. I know there is lots of bearish sentiments here. Lots of people correctly point out that this is not the same math as FAANG products - then they make the jump that it must be bad. But - my guess is these companies end up with margins better than Tesla (modern manufacture…

The difference is the money printer right now only prints for ~6 months before it needs to be replaced with an even more expensive printer.

And if you ever stop/step off the treadmill and jack up prices to reach profitability, a new upstart without your sunk costs will immediately create a 99% solution and start competing with you. Or more like hundreds of competitors. Like we've seen with Karpathy & Murati, any engineer with pedigree working on the frontline models can easily raise billions to compete with them.

Expect the trend to pick up as the pool of engineers who can create usable LLMs from scratch increases through knowledge/talent diffusion.

Re: Are OpenAI and Anthropic losing money on inference?

#88

Post by FarMcKon »

JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

"This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage payment, not because they’re negative on operating costs."

Exactly the analogy I was going to make. :)

Re: Are OpenAI and Anthropic losing money on inference?

#90

Post by JimDabell »

techpineapple wrote: post #79
Aurornis wrote: post #64

Earlier quoted context omitted.

Exactly. All of the claims that OpenAI is losing money on every request are wrong. OpenAI hasn’t even unlocked all of their possible revenue opportunities from the free tier such as ads (like Google search), affiliate links, and other services. There’s also a lot of comments in this thread who want LLM companies to fail for different reasons, so they’re projecting that wish on to imagined unit economics. I’m having f…

Because Sam Altman said so? Sam Altman also said this: https://xcancel.com/sama/status/1876104315296968813

There is a six month gap between those statements. Inference costs have been plummeting, plans have had tweaked quotas, and usage patterns can change.
Post reply on HN