Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

411–420 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#411

Post by recursivecaveat »

whatshisface wrote: post #251
qrios wrote: post #246

Earlier quoted context omitted.

If multiple cores tries to get the same memory addresses, the MMU feeds only one core, the second one have to whait. Depends on the type of RAM, this will cost a lot of cycles. GPU MMUs can handle multiple line in parallel. But not 10k cores at the same time. The HBM is not able to transfer 3.5TByte sequencial.

Why is that? It seems like multiple cores requesting the same address would be easier for the MMU to fetch for, not harder.

Not necessarily the exact same address (you can fix that in a program anyways with a broadcast tree), but same memory bank. Imagine 1000 trains leaving a small town at the same time, instead of 1000 trains leaving 1000 different towns simultaneously. At some point there are not enough transportation resources to get stuff out of a particular area at the parallelism desired.

Re: Are OpenAI and Anthropic losing money on inference?

#412

Post by jrflowers »

dcre wrote: post #402
jrflowers wrote: post #394

Earlier quoted context omitted.

Ed Zitron: I don’t think OpenAI will become profitable The link you posted: I think it is very plausible that it will be hard for OpenAI to become profitable

Are you referring to the post where I listed 4 claims and marked one ridiculous, one wrong, one unlikely, and one plausible? He is not wrong about everything. For example, after Sam Altman said in January that OpenAI would introduce a model picker, Zitron was able to predict in March that OpenAI would introduce a model picker. And he was right about that.

Yes. In this thread about the profitability (or lack thereof) of OpenAI’s business model, I pointed out the part where you appeared to agree with Ed Zitron about the profitability (or lack thereof) of OpenAI’s business model. Like it seems like all of those posts were pretty clearly motivated by wanting to poke holes in Zitron’s criticism, their lack of profitability (which is central to his criticism) is where you declined to push back with any real argument.

Re: Are OpenAI and Anthropic losing money on inference?

#413

Post by chillee »

martinald wrote: post #409
chillee wrote: post #405

Earlier quoted context omitted.

No. In some sense, the article comes to the right conclusion haha. But it's probably >100x off on its central premise about output tokens costing more than input.

Thanks for the correction (author here). I'll update the article - very fair point on compute on input tokens which I messed up. Tbh I'm pleased my napkin math was only 7x off the laws of physics :). Even rerunning the math on my use cases with way higher input token cost doesn't change much though.

The 32 parallel sequences is also arbitrary and significantly changes your conclusions. For example, if they run with 256 parallel sequences then that would result in a 8x cheaper factor in your calculations for both prefill and decode.

The component about requiring long context lengths to be compute-bound for attention is also quite misleading.

Re: Are OpenAI and Anthropic losing money on inference?

#414

Post by dgfitz »

jacurtis wrote: post #398
BlindEyeHalo wrote: post #303

Earlier quoted context omitted.

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

In a recent episode of Hard Fork podcast, the hosts discussed an on-the-record conversation they had with Sam Altman from OpenAI. They asked him about profitability and he claimed that they are losing money mostly because of the cost of training. But as the model advances, they will train less and less. Once you take training out of the equation he claimed they were profitable based on the cost of serving the trained…

> But as the model advances, they will train less and less.

They sure have a lot of training to do between now and whenever that happens. Rolling back from 5 to whatever was before it is their own admission of this fact.

Re: Are OpenAI and Anthropic losing money on inference?

#415

Post by echelon »

mgh95 wrote: post #383
MontyCarloHall wrote: post #314

Earlier quoted context omitted.

As long as models continue on their current rapid improvement trajectory, retraining from scratch will be necessary to keep up with the competition. As you said, that's such a huge amount of continual CapEx that it's somewhat meaningless to consider AI companies' financial viability strictly in terms of inference costs, especially because more capable models will likely be much more expensive to train. But at some po…

> If the latter, assuming the cost of fine-tuning is a fraction of the cost of training from scratch, the low cost of inference does indeed make a bullish case for these companies. On the other hand, this may also turn into cost effective methods such as model distillation and spot training of large companies (similarly to Deepseek). This would erode the comparative advantage of Anthropic and OpenAI, and result in a…

> model distillation

I like to think this is the end of software moats. You can simply call a foundation model company's API enough times and distill their model.

It's like downloading a car.

Distribution still matters, of course.

Re: Are OpenAI and Anthropic losing money on inference?

#416

Post by jcranmer »

dcre wrote: post #366
citizenpaul wrote: post #304

Earlier quoted context omitted.

There a journalist ed zittron https://www.wheresyoured.at/ That is an openai skeptic. His research if correct says not only is openai unprofitable but it likely never will be. Can't be ,its various finance ratios make early uber, amazon ect look downright fiscally frugal. He is not a tech person for what that means to you.

Zitron is not a serious analyst. https://bsky.app/profile/davidcrespo.bsky.social/post/3lxale... https://bsky.app/profile/davidcrespo.bsky.social/post/3lo22k... https://bsky.app/profile/davidcrespo.bsky.social/post/3lwhhz... https://bsky.app/profile/davidcrespo.bsky.social/post/3lv2dx...

Since only the first one responds to any of Zitron's content that I've actually read, I'll respond only to that one:

It's not responsive at all to Zitron's point. Zitron's broader contention is that AI tools are not profitable because the cost of AI use is too high for users to justify spending money on the output, given the quality of output. And furthermore, he argues that this basic fact is being obscured by lots of shell games around numbers to hide the basic cash flow issue. For example, focusing on cost in terms of cost per token rather than cost per task. And finally, there's an implicit assumption that the AI just isn't getting tremendously better, as might be exemplified by... burning twice as money tokens on the task in the hopes the quality goes up.

And in that context, the response is "Aha, he admits that there is a knob to trade off cost and quality! Entire argument debunked!" The existence of a cost-quality tradeoff doesn't speak to whether or not that line will intersect the quality-value tradeoff. I grant that a lot turns on how good you think AI is and/or will shortly be, and Zitron is definitely a pessimist there.

Re: Are OpenAI and Anthropic losing money on inference?

#417

Post by sanxiyn »

While the broad point that no, OpenAI and Anthropic are not losing money on inference, is almost certainly correct, the model itself is quite incomplete.

If you actually want to know, I recommend Inference economics of language models from Epoch AI, which is probably the best public model as of 2025-06.

https://arxiv.org/abs/2506.04645

Re: Are OpenAI and Anthropic losing money on inference?

#418

Post by jsnell »

trilogic wrote: post #317
_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

I have to disagree. The biggest cost is still energy consumption, water and maintenance. Not to mention, to keep up with the rivals in incredibly high tempo (so offering billions like Meta recently). Then the cost of hardware that is equal to Nvidia skyrocketing shares :) No one should dare to talk about profit yet. Now is time to grab the market, invest a lot and work hard, hopping for a future profit. The equation…

The capital costs for the GPU are an order of magnitude larger than the energy consumption. It doesn't matter whether the GPUs are used for training or inference.

Back of the envelope: $25k GPU amortized over 5 years is $5k/year. A 500W GPU run at full power uses 4.5MWh; at $0.15/kWh the electricity costs $650/year.

The other operating costs you suggest have to be even smaller.

Re: Are OpenAI and Anthropic losing money on inference?

#419

Post by jcranmer »

dcre wrote: post #284
overgard wrote: post #248

Earlier quoted context omitted.

In terms of sources, I would trust Zitron a lot more than Altman or Amodei. To be charitable, those CEOs are known for their hyperbole and for saying whatever is convenient in the moment, but they certainly aren't that careful about being precise or leaving out inconvenient details. Which is what a CEO should do, more or less, but, I wouldn't trust their word on most things.

I agree we should not take CEOs at their word, we have to think about whether what they're saying is more likely to be true than false given other things we know. But to trust Zitron on anything is ridiculous. He is not a source at all: he knows very little, does zero new reporting, and frequently contradicts himself in his frenzy to believe the bubble is about to pop any time now. A simple example: claiming both tha…

Those two statements are not contradictory, and thinking that they are belies a pretty fundamental misunderstanding of his basic thesis.

The first statement is one about the present value of AI. The second statement is about their belief of the future value of AI.

Re: Are OpenAI and Anthropic losing money on inference?

#420

Post by jsnell »

overgard wrote: post #263
jsnell wrote: post #58

Earlier quoted context omitted.

For the top few providers, the training is getting amortized over absurd amount of inference. E.g. Google recently mentioned that they processed 980T tokens over all surfaces in June 2025. The leaked OpenAI financial projections for 2024 showed about equal amount of money spent on training and inference. Amortizing the training per-query really doesn't meaningfully change the unit economics. > Fact remains when all c…

Assuming users accept those ads. Like, would they make it clear with a "sponsored section", or would they just try to worm it into the output? I could see a lot of potential ways that users reject the ad service, especially if it's seen to compromise the utility or correctness of the output.

Billions of people use Google, YouTube, Facebook, Tiktok, Instagram, etc and accept the ads. Getting similar ad rates would make OpenAI fabulously profitable. They have no need to start with ad formats that might be rejected by users. Even if that were the intended endgame, you'd want to boil the frog for years.
Post reply on HN