[deleted]
Are OpenAI and Anthropic losing money on inference?
331–340 of 495 posts
Re: Are OpenAI and Anthropic losing money on inference?
#332Earlier quoted context omitted.
The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…
>Also, in Nike's case, as they grow they get better at making more shoes for cheaper. This is clearly the case for models as well. Training and serving inference for GPT4 level models is probably > 100x cheaper than they used to be. Nike has been making Jordan 1's for 40+ years! OpenAI would be incredibly profitable if they could live off the profit from improved inference efficiency on a GPT4 level model!
>>This is clearly the case ... probably
>>OpenAI would be incredibly profitable if they could live off the profit from improved inference efficiency on a GPT4 level model!
If gpt4 was basically free money at this point it's real weird that their first instinct was to cut it off after gpt5
Re: Are OpenAI and Anthropic losing money on inference?
#333Earlier quoted context omitted.
The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…
But new models to date have cost more than the previous ones to create, often by an order of magnitude, so the shoe metaphor falls apart. A better metaphor would be oil and gas production, where existing oil and gas fields are either already finished (i.e. model is no longer SOTA -- no longer making a return on investment) or currently producing (SOTA inference -- making a return on investment). The key similarity wi…
exactly: it’s like making shoes if you’re really bad at making shoes :)
Re: Are OpenAI and Anthropic losing money on inference?
#334Earlier quoted context omitted.
ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…
The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…
I believe better analogy is CPU development on next process node.
each node is much more expensive to design for, but when you finally have it you basically print money.
and of course you always have to develop next more powerful and power efficient CPU to keep competitive
Re: Are OpenAI and Anthropic losing money on inference?
#335Earlier quoted context omitted.
Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.
I suspect we've already reached the point with models at the GPT5 tier where the average person will no longer recognize improvements and this model can be slightly improved at slow intervals and indeed run for years. Meanwhile research grade models will still need to be trained at massive cost to improve performance on relatively short time scales.
Whenever someone has complained to me about issues they are having with ChatGPT on a particular question or type of question, the first thing I do is ask them what model they are using. So far, no one has ever known offhand what model they were using, nor were not aware there are more models!
If you understand there are multiple models from multiple providers, some of those models are better at certain things than others, and how you can get those models to complete your tasks, you are in the top 1% (probably less) of LLM users.
Re: Are OpenAI and Anthropic losing money on inference?
#336I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…
can you share the model?
Re: Are OpenAI and Anthropic losing money on inference?
#337I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…
I agree that you could get to high margins, but I think the modeling holds only if you're an AI lab operating at scale with a setup tuned for your model(s). I think the most open study on this one is from the DeepSeek team: https://github.com/deepseek-ai/open-infra-index/blob/main/20...
For others, I think the picture is different. When we ran benchmarks on DeepSeek-R1 on 8x H200 SXM using vLLM, we got up to 12K total tok/s (concurrency 200, input:output ratio of 6:1). If you're spiking up 100-200K tok/s, you need a lot of GPUs for that. Then, the GPUs sit idle most of the time.
I'll read the blog post in more detail, but I don't think the following assumptions hold outside of AI labs.
* 100% utilization (no spikes, balanced usage between day/night or weekdays) * Input processing is free (~$0.001 per million tokens) * DeepSeek fits into H100 cards in a way that network isn't the bottleneck
Re: Are OpenAI and Anthropic losing money on inference?
#338Earlier quoted context omitted.
OK, but you are a person, not a company. "You" are not taking the share away.
"I'm digging a trench" "No you're not, WE are digging a trench!" Yes fine, but "I am as well". Sheesh. Also I, personally, do and lead the work of taking the wallet share. So I will stick with "I" and would accept any of my team saying the same.
Well, at least your attitude has made it obvious who you work for now. ;)
Re: Are OpenAI and Anthropic losing money on inference?
#339https://www.axios.com/2025/08/15/sam-altman-gpt5-launch-chat... quotes Sam Altman saying: > Most of what we're building out at this point is the inference [...] We're profitable on inference. If we didn't pay for training, we'd be a very profitable company.
ICYMI, Amodei said the same in much greater detail: "If you consider each model to be a company, the model that was trained in 2023 was profitable. You paid $100 million, and then it made $200 million of revenue. There's some cost to inference with the model, but let's just assume, in this cartoonish cartoon example, that even if you add those two up, you're kind of in a good state. So, if every model was a company,…
>> If we didn't pay for training, we'd be a very profitable company.
> ICYMI, Amodei said the same
No. He says that even paying for training a model is profitable. It makes more revenue that it costs - all things considered. A much stronger claim.
Re: Are OpenAI and Anthropic losing money on inference?
#340Earlier quoted context omitted.
Well, only if the one training model continued to function as a going business. Their amortization window for the training cost is 2 months or so. They can't just keep that up and collect $. They have to build the next model, or else people will go to someone else.
Why two months? It was almost a year between Claude 3.5 and 4. (Not sure how much it costs to go from 3.5 to 3.7.)
Even being generous, and saying it's a year, most capital expenditures depreciate over a period of 5-7 years. To state the obvious, training one model a year is not a saving grace