Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

451–460 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#451

Post by fallmonkey »

atq2119 wrote: post #408
fallmonkey wrote: post #308

The estimation for output token is too low since one reasoning-enabled response can burn through thousands of output tokens. Also low for input tokens since in actual use there're many context (memory, agents.md, rules, etc) included nowadays.

When using APIs, you pay for reasoning tokens like you do for actual outputs. So, the estimation on a per-token basis is not affected by reasoning. What reasoning affects is the ratio of input to output tokens, and since input tokens are cheaper, that may well affect the economics in the end.

Correct, and with reasoning, the ratio is totally off. As others have pointed out, actual usage is way higher (much more than 3-5x) than the estimation in the article, which is probably for very trivial users.

Re: Are OpenAI and Anthropic losing money on inference?

#452

Post by resonious »

Aeolun wrote: post #425
chillee wrote: post #395

This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it's 1.44 million input tokens per second * 37 billion active params * 2 (FMA) / 8 [GPUs per instance] = 13 Petaflops per second. That's approximately 7x absolutely peak FLOPS on the hardware. Obviously, that's impossible. There's many other…

As much as I appreciate you saying the math is wrong, it doesn’t really help me adjust my expectations unless you provide correct numbers as well.

Right. Now I want to know if they're really losing money or not.

Re: Are OpenAI and Anthropic losing money on inference?

#453

Post by hnfsfdsd »

mindwok wrote: post #428
dgfitz wrote: post #414

Earlier quoted context omitted.

> But as the model advances, they will train less and less. They sure have a lot of training to do between now and whenever that happens. Rolling back from 5 to whatever was before it is their own admission of this fact.

I think that actually proves the opposite. People wanted an old model, not a new one, indicating that for that user base they could have just... not trained a new model.

If people want old models, they can go to any of the competitor's , deepseek, claud, opensources, etc... That's not good news for OpenAI.

Re: Are OpenAI and Anthropic losing money on inference?

#454

Post by johnnypangs »

chillee wrote: post #395

This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it's 1.44 million input tokens per second * 37 billion active params * 2 (FMA) / 8 [GPUs per instance] = 13 Petaflops per second. That's approximately 7x absolutely peak FLOPS on the hardware. Obviously, that's impossible. There's many other…

As one of those people who doesn’t really understand llms, does anyone have any recommendations to better my understanding of them?

Re: Are OpenAI and Anthropic losing money on inference?

#455

Post by a333999 »

Why is everyone so mean? I believe the article delivered on its headline.

The author showed that no -large llm providers do not loose money on inference. Model training is not accounted for because that was not the point.

I personally felt that the maths calculations was a bit redundant, since after the maths part the same numbers are taken from open router pricing. But I think it is a matter of presentation.

I would have shown OR pricing first and then did the math. In that way it would have been as insightful, since it still showed that model providers do also make money and the reader would not have felt that he did maths he could have avoided :)

So, thanks to the author! Good job.

The feedback fro hn is overly harsh. Idk; It makes me sad how mean people have become. I guess the world is not in a great spot, but taking out anger on strangers will only make it worse.

Re: Are OpenAI and Anthropic losing money on inference?

#456

Post by anothernewdude »

jacurtis wrote: post #398
BlindEyeHalo wrote: post #303

Earlier quoted context omitted.

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

In a recent episode of Hard Fork podcast, the hosts discussed an on-the-record conversation they had with Sam Altman from OpenAI. They asked him about profitability and he claimed that they are losing money mostly because of the cost of training. But as the model advances, they will train less and less. Once you take training out of the equation he claimed they were profitable based on the cost of serving the trained…

Unfortunately for those companies, their APIs are a commodity, and are very fungible. So they'll need to keep training or be replaced with whichever competitor will. This is an exercise in attrition.

Re: Are OpenAI and Anthropic losing money on inference?

#457

Post by PeterStuer »

mindwok wrote: post #428
dgfitz wrote: post #414

Earlier quoted context omitted.

> But as the model advances, they will train less and less. They sure have a lot of training to do between now and whenever that happens. Rolling back from 5 to whatever was before it is their own admission of this fact.

I think that actually proves the opposite. People wanted an old model, not a new one, indicating that for that user base they could have just... not trained a new model.

That is for a very specific class of usecases. If they would turn up the sycophancy on the new model, those people would not call for the old onee.

The reasoning here is off. It is like saying new game development is nearly over as some people keep playing old games.

My feeling: we've yet barely scrarched the surface on the milage we can get out of even today's frontier models, but we are just at the beginning of a huge runway for improved models and architectures. Watch this space.

Re: Are OpenAI and Anthropic losing money on inference?

#458

Post by johnnienaked »

But there's no way those "costs" are real because companies are currently losing money on purpose to accumulate market share and stay ahead of their competitors, similar to Amazon selling diapers at a loss to put diapers.com out of business and then jacking the prices.

So the true cost could be 10x as much as stated, we have no idea.

Re: Are OpenAI and Anthropic losing money on inference?

#459

Post by citizenpaul »

dcre wrote: post #366
citizenpaul wrote: post #304

Earlier quoted context omitted.

There a journalist ed zittron https://www.wheresyoured.at/ That is an openai skeptic. His research if correct says not only is openai unprofitable but it likely never will be. Can't be ,its various finance ratios make early uber, amazon ect look downright fiscally frugal. He is not a tech person for what that means to you.

Zitron is not a serious analyst. https://bsky.app/profile/davidcrespo.bsky.social/post/3lxale... https://bsky.app/profile/davidcrespo.bsky.social/post/3lo22k... https://bsky.app/profile/davidcrespo.bsky.social/post/3lwhhz... https://bsky.app/profile/davidcrespo.bsky.social/post/3lv2dx...

Well yes. He is a journalist. Not an analyst. I don't think he is a prophet or anything nor is is a tech person so yeah his details on operation are off. The money business side though he seem to be the only person willing to poke holes in the whole AI thing publicly.

I'm not an AI hater. I genuinely hope it take over every single white collar job that exists. I'm not being sarcastic or hyperbolic. Only then will we be able to re-discuss what society is in a more humane way.

Re: Are OpenAI and Anthropic losing money on inference?

#460

Post by jillesvangurp »

yalogin wrote: post #373

Will these companies ever stop training new models? What does it mean if we get there. Feels like they will have to constantly train and improve the models, not sure what that means either. What ncremental improvements can these models show? Another question is - will it ever become less costly to train? Let to see opinions from someone in the know

There's a difference between model training and model fine tuning. Model training is not something e.g. OpenAI does a lot. The cutoff for gpt 5 was actually quite long a go. They sat on that for almost a year while they fine tuned the model. Finetuning is a lot faster and cheaper than training the full model. Training + finetuning is essentially fixed cost.

And you have to see this in proportion to the revenue. If you charge 20$/month and you have a few tens of millions of paying users and some premium tier users, that generates quite a bit of revenue.

OpenAI recently claimed they have 700 million regular users. I'm not sure how real/accurate that number is, but if one tenth of those pay for it it, that would be 1.4 billion per month coming in. That's excluding higher tiers. And I suspect they are shooting for a much larger market and are going to be nudging people to a bit higher tiers. Some have suggested that employers paying hundreds of dollars per month for AI subscriptions per employee might become normal in some sectors. That's an awful lot of money and I don't think they are done growing.

And of course with that kind of revenue, you can burn some cash on training cost. A few hundred million is basically nothing.

OpenAI has raised tens of billions of money. But they should be making 10-20 billions of revenue per year as well with some healthy growth. And they are showing very little signs of running out of money.

Post reply on HN