From https://www.theverge.com/command-line-newsletter/759897/sam-... , Sam Altman said: > “If we didn’t pay for training, we’d be a very profitable company.”
Or if they had to pay copyright costs. So much pirated data being repackaged and sold.
Are OpenAI and Anthropic losing money on inference?
101–110 of 495 posts
Re: Are OpenAI and Anthropic losing money on inference?
#102If inference is that cheap, why is not even one company profitable yet?
"Why would you reinvest profits back into a business that is extremely profitable, when you have the chance of pulling your money out?"
You are making a joke but reasonably speaking there are a ton of software companies where they kept reinvesting where they should have taken out profit, especially when they are peaking.
Re: Are OpenAI and Anthropic losing money on inference?
#103If inference is that cheap, why is not even one company profitable yet?
Because they're spending it all on training the next model.
That's an argument for why openai and anthropic shouldn't be profitable, but this point is about how also they don't have customers using the models to generate a profit either. Things like cursor, for example. ETA: also note the recent MIT study that found that 95% of LLM pilots at for-profit companies were not producing returns.
Re: Are OpenAI and Anthropic losing money on inference?
#104Earlier quoted context omitted.
The marginal cost is not the salient factor when the model has to be frequently retrained at great cost. Even if the marginal cost was driven to zero, would they profit?
But they don't have to be retained frequently at great cost. Right now they are retrained frequently because everyone is frequently coming out with new models and nobody wants to fall behind. But if investment for AI were to dry up everyone would stop throwing so much money at R&D, and if everyone else isn't investing in new models you don't have to either. The models are powerful as they are, most of the knowledge i…
Like we've seen with Karparthy & Murati starting their own labs, it's to be expected that over the next 5 years, hundreds of engineers & researchers at the bleeding edge will quit and start competing products. They'll reliably raise $1b to $5b in weeks, too. And it's logical: for an investor, a startup founded by a Tier 1 researcher will more reliably 10-100x your capital, vs. Anthropic & OpenAI that are already at >$250b+.
This talent diffusion guarantees that OpenAI and Anthropic will have to keep sinking in ever more money to stay at the bleeding edge, or upstarts like DeepSeek and incumbents like Meta will simply outspend you/hire away all the Tier 1 talent to upstage you.
The only companies that'll reliably print money off AI are TSMC and NVIDIA because they'll get paid either way. They're selling shovels and even if the gold rush ends up being a bust, they'll still do very well.
Re: Are OpenAI and Anthropic losing money on inference?
#105The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on…
Well he asked some AI to do the math for him probably
Re: Are OpenAI and Anthropic losing money on inference?
#106The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on…
Your calculations make no sense. Why are you loading the model for each token independently? You can process all the input tokens at the same time as long as they can fit in memory.
You are doing the calculation as they were output tokens on a single batch, it would not make sense even in the decode phase.
Re: Are OpenAI and Anthropic losing money on inference?
#107The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on…
> 37e9 bytes/token
This doesn't quite sound right...isn't a token just a few characters?
Re: Are OpenAI and Anthropic losing money on inference?
#108Earlier quoted context omitted.
The marginal cost is not the salient factor when the model has to be frequently retrained at great cost. Even if the marginal cost was driven to zero, would they profit?
But they don't have to be retained frequently at great cost. Right now they are retrained frequently because everyone is frequently coming out with new models and nobody wants to fall behind. But if investment for AI were to dry up everyone would stop throwing so much money at R&D, and if everyone else isn't investing in new models you don't have to either. The models are powerful as they are, most of the knowledge i…
> But if investment for AI were to dry up everyone would stop throwing so much money at R&D, and if everyone else isn't investing in new models you don't have to either
IF.
If you do stagnate for years someone will eventually decide to invest and beat you. Intel has proven so.
Re: Are OpenAI and Anthropic losing money on inference?
#109This is a great article, but it doesn't appear to model H100 downtime in the $2/hr costs. It assumes that OpenAI and Anthropic can match demand for inference to their supply of H100s perfectly, 24/7, in all regions. Maybe you could argue that the idle H100s are being used for model training - but that's different to the article's argument that inference is economically sustainable in isolation.
Not really, that is why they sell Batch API at considerably lower costs than the normal API.
There are also probably all kinds of enterprise deals that they are okay with high latency (> hours) that they do beyond the PAYG batch APIs
Re: Are OpenAI and Anthropic losing money on inference?
#110Ok, one issue I have with this analysis is the breakdown between input and output tokens. I'm the kind of person who spend most of my chat asking questions, so I might only use 20ish input tokens per prompt, where Gemini is having to put out several hundred, which would seem to affect the economics quite a bit