Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

271–280 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#271

Post by skybrian »

hirako2000 wrote: post #164
noodletheworld wrote: post #145

Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap,…

Imo the article is totally off the mark since it assumes users on average do not go over th 1M tokens per day. Afaik openai doesn't enforce a daily quota even on the $20 plans unless the platform is under pressure. Since I often consume 20M token per day, one can assume many would use far more than the 1M tokens assumed in the article's calculations.

Meanwhile, I don’t use ChatGPT at all on a median day. I use it in occasional bursts when researching something.

Re: Are OpenAI and Anthropic losing money on inference?

#272

Post by jumploops »

LZ_Khan wrote: post #265
dcre wrote: post #219

Earlier quoted context omitted.

ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…

Also Amodei has an assumption that a 100m model will make 200m of revenue but a 1B model will make 2B of revenue. Does that really hold up? There's no phenomenon that prevents them from only making 200m of revenue off a $1B model.

GPT-4.5 has entered the chat..

Re: Are OpenAI and Anthropic losing money on inference?

#273

Post by idiotsecant »

dragonwriter wrote: post #177
noodletheworld wrote: post #145

Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap,…

> but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices There are multiple API providers offering models at dirt cheap prices, enough so that there is at least one well-known API provider that is an aggreggator of other API providers that offers lots of models at $0. > The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1?…

How is this possible? I imagine someone is finding some value in the prompts themselves but this cant possibly be paying for itself.

Re: Are OpenAI and Anthropic losing money on inference?

#274

Post by CjHuber »

hirako2000 wrote: post #157
CjHuber wrote: post #146

Earlier quoted context omitted.

The reason it shook the market at least was because of the claim that its training cost was 5 million.

That' what the buzz focused on, strange as we don't actually know what it cost them. While inference optimization is a fact and is even more impactful since training costs benefit from economics of scale.

I don't think that's strange at all, it's a much more palatable narrative for the mass who doesn't know what inference and training is and who think having conversations=training

Re: Are OpenAI and Anthropic losing money on inference?

#275

Post by snowwrestler »

Not wishing to do a shallow dismissal here, but I always assumed AI must be profitable on inference otherwise no one would pursue it as a business given how expensive the training is.

It seems sort of like wondering if a fiber ISP is profitable per GB bandwidth. Of course it is; the expensive part is getting the fiber to all the homes. So the operations must be profitable or there is simply no business model possible.

Re: Are OpenAI and Anthropic losing money on inference?

#276

Post by _sword »

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to be considered... yikes

Re: Are OpenAI and Anthropic losing money on inference?

#277

Post by dcre »

prasadjoglekar wrote: post #234
dcre wrote: post #229

Earlier quoted context omitted.

The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.

Well, only if the one training model continued to function as a going business. Their amortization window for the training cost is 2 months or so. They can't just keep that up and collect $. They have to build the next model, or else people will go to someone else.

Why two months? It was almost a year between Claude 3.5 and 4. (Not sure how much it costs to go from 3.5 to 3.7.)

Re: Are OpenAI and Anthropic losing money on inference?

#278

Post by dcre »

ToucanLoucan wrote: post #235
dcre wrote: post #229

Earlier quoted context omitted.

The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.

So is OpenAI capable of not making a new model at some point? They've been training the next model continuously as long as they've existed AFAIK. Our software house spends a lot on R&D sure, but we're still incredibly profitable all the same. If OpenAI is in a position where they effectively have to stop iterating the product to be profitable, I wouldn't call that a very good place to be when you're on the verge of h…

I think at that point there is strong financial pressure to figure out how to continuously evolve models instead of changing new ones, for example by building models out of smaller modules that can be trained individually and swapped out. Jeff Dean and Noam Shazeer talked about that a bit in their interview with Dwarkesh: https://www.dwarkesh.com/p/jeff-dean-and-noam-shazeer

Re: Are OpenAI and Anthropic losing money on inference?

#279

Post by jstummbillig »

overgard wrote: post #245
thatguysaguy wrote: post #201

Earlier quoted context omitted.

Why would you think that deepseek is more efficient than gpt-5/Claude 4 though? There's been enough time to integrate the lessons from deepseek.

Because to make GPT-5 or Claude better than previous models, you need to do more reasoning which burns a lot more tokens. So, your per-token costs may drop, but you may also need a lot more tokens.

GPT-5 can be configured extensively. Is there any point at which any configuration of GPT-5 that offers ~DeepSeek level performance is more expensive than DeepSeek per token?

Re: Are OpenAI and Anthropic losing money on inference?

#280

Post by lawlessone »

_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

Does that include legal fights and potential payouts to artists and writers whose work was used without permission?

Can anyone explain why it's not allowed to compensate the creators of the data?

Post reply on HN