Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

141–150 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#141

Post by martinald »

moduspol wrote: post #97

This kind of presumes you're just cranking out inference non-stop 24/7 to get the estimated price, right? Or am I misreading this? In reality, presumably they have to support fast inference even during peak usage times, but then the hardware is still sitting around off of peak times. I guess they can power them off, but that's a significant difference from paying $2/hr for an all-in IaaS provider. I'm also not sure w…

Yes. But these are on demand prices, so you could just turn them off when loads are less.

But there is no way that OpenAI should be more expensive than this. The main cost is the capex of the H100s, and if you are buying 100k at a time you should be getting a significant discount off list price.

Re: Are OpenAI and Anthropic losing money on inference?

#142

Post by mutkach »

thatguysaguy wrote: post #112
ekelsen wrote: post #98

The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on…

37 billion bytes per token? Edit: Oh assuming this is an estimate based on the model weights moving fromm HBM to SRAM, that's not how transformers are applied to input tokens. You only have to do move the weights for every token during generation, not during "prefill". (And actually during generation you can use speculative decoding to do better than this roofline anyways).

There's also an estimation of how much a KV cache grows with each subsequent token. That would be roughly ~MBs/token. I think that would be the bottleneck

Re: Are OpenAI and Anthropic losing money on inference?

#144

Post by est31 »

Barbing wrote: post #119
JimDabell wrote: post #85

Earlier quoted context omitted.

There’s no mention of that in this article about it: https://archive.is/wZslL They quote him as saying inference is profitable and end it at that. Are you saying that the COO corrected him at the dinner, or on the podcast? Which podcast was it?

From a journalist at the dinner: “I think that tends to end poorly because as demand for your service grows, you lose more and more money. Sam Altman actually addressed this at dinner. He was asked basically, are you guys losing money every time someone uses ChatGPT? And it was funny. At first, he answered, no, we would be profitable if not for training new models. Essentially, if you take away all the stuff, all the…

GPT-5 was I suppose their attempt to make a product that provides as good metrics as their earlier products.

Uber doesn't really compare, as they had existing competition from taxi companies that they first had to/have to destroy. And cars or fuel didn't get 10x cheaper over the time of Uber's existence, but I'm sure that they still can optimize a lot for efficiency.

I'm more worried about OpenAIs capability to build a good moat. Right now it seems that each success is replicated by the competing companies quickly. Each month there is a new leader in the benchmarks. Maybe the moat will be the data in the end, i.e. there is barriers nowadays to crawl many websites that have lots of text. Meanwhile they might make agreements with the established AI players, maybe some of those agreements will be exclusive. Not just for training but also for updating wrt world news.

Re: Are OpenAI and Anthropic losing money on inference?

#145

Post by noodletheworld »

Huh.

I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices?

The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1?

Surely if its this cheap, and we're talking massive margins according to this, I should be able to get a cheap / run my own 600B param model.

Am I missing something?

It seems that reality (ie. the absence of people actually doing things this cheap) is the biggest critic of this set of calculations.

Re: Are OpenAI and Anthropic losing money on inference?

#146

Post by CjHuber »

sc68cal wrote: post #125

This whole article is built off using DeepSeek R1, which is a huge premise that I don't think is correct. DeepSeek is much more efficient and I don't think it's a valid way to estimate what OpenAI and Anthropic's costs are. https://www.wheresyoured.at/deep-impact/ Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.

The reason it shook the market at least was because of the claim that its training cost was 5 million.

Re: Are OpenAI and Anthropic losing money on inference?

#147

Post by nativeit »

EcommerceFlow wrote: post #94
techpineapple wrote: post #84

Earlier quoted context omitted.

> They have a service which understands a users question/needs 100x better than a traditional Google search does. Source?

A lifetime of using Google and 4 years of using LLMs.

…is a great counter-example of a “source”.

It’s not like the product at-hand is relevant to data analysis or anything, amirite?

Re: Are OpenAI and Anthropic losing money on inference?

#148

Post by CjHuber »

Topfi wrote: post #114
gitremote wrote: post #45

These numbers are off. > $20/month ChatGPT Pro user: Heavy daily usage but token-limited ChatGPT Pro is $200/month and Sam Altman already admitted that OpenAI is losing money from Pro subscriptions in January 2025: "insane thing: we are currently losing money on openai pro subscriptions! people use it much more than we expected." - Sam Altman, January 6, 2025 https://xcancel.com/sama/status/1876104315296968813

That doesn't seem compatible with what he stated more recently: > We're profitable on inference. If we didn't pay for training, we'd be a very profitable company. Source: https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chat... His possible incentives and the fact OpenAI isn't a public company simply make it hard for us to gauge which of these statements is closer to the truth.

Does anybody really think in this current time that what a CEO says has anything to do with reality and not just with hyping up ala elon recipe

Re: Are OpenAI and Anthropic losing money on inference?

#149

Post by metalliqaz »

Topfi wrote: post #114
gitremote wrote: post #45

These numbers are off. > $20/month ChatGPT Pro user: Heavy daily usage but token-limited ChatGPT Pro is $200/month and Sam Altman already admitted that OpenAI is losing money from Pro subscriptions in January 2025: "insane thing: we are currently losing money on openai pro subscriptions! people use it much more than we expected." - Sam Altman, January 6, 2025 https://xcancel.com/sama/status/1876104315296968813

That doesn't seem compatible with what he stated more recently: > We're profitable on inference. If we didn't pay for training, we'd be a very profitable company. Source: https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chat... His possible incentives and the fact OpenAI isn't a public company simply make it hard for us to gauge which of these statements is closer to the truth.

> If we didn't pay for training

it is comical that something like this was even uttered in the conversation. It really shows how disconnected the tech sector is from the real world.

Imagine Intel CEO saying "If we didn't have to pay for fabs, we'd be a very profitable company." Even in passing. He'd be ridiculed.

Re: Are OpenAI and Anthropic losing money on inference?

#150

Post by brokencode »

noodletheworld wrote: post #145

Huh. I feel oddly skeptical about this article; I can't specifically argue the numbers, since I have no idea, but... there are some decent open source models; they're not state of the art, but if inference is this cheap then why aren't there multiple API providers offering models at dirt cheap prices? The only cheap-ass providers I've seen only run tiny models. Where's my cheap deepseek-R1? Surely if its this cheap,…

I also have no idea on the numbers. But I do know that these same companies are pouring many billions of dollars into training models, paying very expensive staff, and building out infrastructure. These costs would need to be factored in to come up with the actual profit margins.
Post reply on HN