Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

281–290 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#281

Post by drob518 »

dcre wrote: post #229
drob518 wrote: post #223

Earlier quoted context omitted.

Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and therefore basically meaningless.

The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.

So,if they stopped training they’d be profitable? Only in some incremental sense, ignoring all sunk costs.

Re: Are OpenAI and Anthropic losing money on inference?

#282

Post by ergocoder »

lawlessone wrote: post #280
_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

Does that include legal fights and potential payouts to artists and writers whose work was used without permission? Can anyone explain why it's not allowed to compensate the creators of the data?

Of course not. Those usually wouldn't be considered "margin".

Another similar example is R&D and development by engineers aren't considered in margin either.

Re: Are OpenAI and Anthropic losing money on inference?

#283

Post by ekelsen »

GaggiX wrote: post #106
ekelsen wrote: post #98

The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on…

Your calculations make no sense. Why are you loading the model for each token independently? You can process all the input tokens at the same time as long as they can fit in memory. You are doing the calculation as they were output tokens on a single batch, it would not make sense even in the decode phase.

Then the right calculation is to use FLOPs not bandwidth like they did.

Re: Are OpenAI and Anthropic losing money on inference?

#284

Post by dcre »

overgard wrote: post #248
dcre wrote: post #218

Earlier quoted context omitted.

What are we meant to take away from the 8000 word Zitron post? In any case, here is what Anthropic CEO Dario Amodei said about DeepSeek: "DeepSeek produced a model close to the performance of US models 7-10 months older, for a good deal less cost (but not anywhere near the ratios people have suggested)" "DeepSeek-V3 is not a unique breakthrough or something that fundamentally changes the economics of LLM’s; it’s an e…

In terms of sources, I would trust Zitron a lot more than Altman or Amodei. To be charitable, those CEOs are known for their hyperbole and for saying whatever is convenient in the moment, but they certainly aren't that careful about being precise or leaving out inconvenient details. Which is what a CEO should do, more or less, but, I wouldn't trust their word on most things.

I agree we should not take CEOs at their word, we have to think about whether what they're saying is more likely to be true than false given other things we know. But to trust Zitron on anything is ridiculous. He is not a source at all: he knows very little, does zero new reporting, and frequently contradicts himself in his frenzy to believe the bubble is about to pop any time now. A simple example: claiming both that "AI is very little of big tech revenue" and "Big tech has no other way to show growth other than AI hype". Both are very nearly direct quotes.

Re: Are OpenAI and Anthropic losing money on inference?

#285

Post by red2awn »

caminanteblanco wrote: post #110

Ok, one issue I have with this analysis is the breakdown between input and output tokens. I'm the kind of person who spend most of my chat asking questions, so I might only use 20ish input tokens per prompt, where Gemini is having to put out several hundred, which would seem to affect the economics quite a bit

It also didn't take into account a lot of the new models are reasoning models which spits out a lot of output tokens.

Re: Are OpenAI and Anthropic losing money on inference?

#286

Post by Night_Thastus »

lawlessone wrote: post #280
_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

Does that include legal fights and potential payouts to artists and writers whose work was used without permission? Can anyone explain why it's not allowed to compensate the creators of the data?

It's already questionable if anyone can make it profitable once you account for all the costs. Why do you think they try to squash the legal concerns so hard? If they move fast and stick their fingers in their ears, they can just steal whatever the want.

Re: Are OpenAI and Anthropic losing money on inference?

#287

Post by dcre »

gmerc wrote: post #231
dcre wrote: post #218

Earlier quoted context omitted.

What are we meant to take away from the 8000 word Zitron post? In any case, here is what Anthropic CEO Dario Amodei said about DeepSeek: "DeepSeek produced a model close to the performance of US models 7-10 months older, for a good deal less cost (but not anywhere near the ratios people have suggested)" "DeepSeek-V3 is not a unique breakthrough or something that fundamentally changes the economics of LLM’s; it’s an e…

Grok 3.5: 400M training run DeepSeek R1: 5M training run Released around the same time, marginal performance difference.

I suspect that says more about Grok than anything else.

Re: Are OpenAI and Anthropic losing money on inference?

#288

Post by drob518 »

gomox wrote: post #237
drob518 wrote: post #223

Earlier quoted context omitted.

Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and therefore basically meaningless.

I can't imagine the hoops an accountant would have to go through to argue training cost is COGS. In the most obvious stick-figures-for-beginners interpretation, as in, "If I had to explain how a P&L statement works to an AI engineer", training is R&D cost and inference cost is COGS.

I wasn’t using COGS in a GAAP sense, but rather as a synonym for unspecified “costs.” My bad. I suppose you would classify training as development and ongoing datacenter and GPU costs as actual GAAP COGS. My point was, if all you focus on is revenue and ignore the costs of creating your business and keeping it running, it’s pretty easy for any business to be “profitable.”

Re: Are OpenAI and Anthropic losing money on inference?

#289

Post by renjimen »

meshugaas wrote: post #262
dcre wrote: post #219

Earlier quoted context omitted.

ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…

The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…

But new models to date have cost more than the previous ones to create, often by an order of magnitude, so the shoe metaphor falls apart.

A better metaphor would be oil and gas production, where existing oil and gas fields are either already finished (i.e. model is no longer SOTA -- no longer making a return on investment) or currently producing (SOTA inference -- making a return on investment). The key similarity with AI is new oil and gas fields are increasingly expensive to bring online because they are harder to make economical than the first ones we stumbled across bubbling up in the desert, and that's even with technological innovation. That is to say, the low hanging fruit is long gone.

Re: Are OpenAI and Anthropic losing money on inference?

#290

Post by pegasus »

meshugaas wrote: post #262
dcre wrote: post #219

Earlier quoted context omitted.

ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…

The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…

If you're going to use shoes as the metaphor, a model would be more like a shoe factory. A shoe would be a LLM answer, i.e. inference. In which case it totally makes sense to consider each factory as an autonomous economic unit, like a company.
Post reply on HN