Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why current LLM costs are not sustainable

aditya.patadia.org

Post reply on HN
Print view View original

181–190 of 216 posts

Re: Why current LLM costs are not sustainable

#181

Post by piva00 »

hparadiz wrote: post #151
FinnLobsien wrote: post #47

Earlier quoted context omitted.

I think the problem is that the companies mostly don't make money, period. They may have better unit economics on underused subscriptions, but I don't see a world in which OAI/Anthropic don't heavily tighten the screws in the future. Right now it's silly to default to frontier models, but it won't bankrupt your company. I believe in the short-medium term future, we'll need to be more deliberate about model choices. I…

[flagged]

You can save it to your favourites, no need to comment at all if it's not going to add to the conversation.

Re: Why current LLM costs are not sustainable

#182

Post by LUmBULtERA »

jijijijij wrote: post #172
LUmBULtERA wrote: post #170

Earlier quoted context omitted.

Day 2 the craftsman has not made up for the investment/loss of their equipment. Not a useful example.

Sorry, I don't understand what you are trying to say.

The craftsman, who may otherwise be profitable, also has investment costs that cause them to show a loss for some time.

Re: Why current LLM costs are not sustainable

#183

Post by g-b-r »

fragmede wrote: post #137
Hamuko wrote: post #100

Earlier quoted context omitted.

I don't understand this argument. How does it make the subscription any less subsidised if the losses are only because developing the product is just so darn expensive? Feels like arguing that it's not clear if Bugatti's losses came from selling the Veyron instead of designing and developing the Veyron.

The equivalent is when Amazon was running a loss because they were spending all their money on building warehouses. It exactly make sense, but that's the argument.

How often does Amazon have to rebuild their warehouses?

Re: Why current LLM costs are not sustainable

#184

Post by jillesvangurp »

bflesch wrote: post #36
jillesvangurp wrote: post #21

Curren prices will come down. There is a lot of potential for optimization. Energy efficiency, energy generation, self hosting, model size and specialization. Etc. Rught now the state of the art is powering data centers with gas powered turbine generators. That's not very efficient.

Of course, but will the AI startups with their SaaS business model survive?

Not all of them. The mobile internet is a good analogy. Lots of telekom providers that invested heavily in infrastructure and then had to earn that back via subscriptions. But in the end that became a race to the bottom and a lot of their attempts to create walled gardens completely flopped when Apple and others came along. MMS got replaced by iphoto, instagram, etc. SMS got replaced by messaging clients. And so on.

I don't believe in the model where OpenAI or Anthropic will own the whole value chain. They'll try and probably fail. They'll own lots of infrastructure and there's going to be a shortage of that for some time. But most of the value creation will happen upstream from them.

Re: Why current LLM costs are not sustainable

#185

Post by Yizahi »

usef- wrote: post #168
Yizahi wrote: post #156

Earlier quoted context omitted.

Anyone can claim they are profitable, simply by reclassifying their expenses as some other thing or shuffling them to separate corporate structure. Until we will real financial audit, the CEOs claims are just a hot air.

OpenAI's leaked documents also said OpenAI was profitable on inference. The small resellers of open models have nowhere near the resources to optimise their models or inference and yet usually have a lower cost, why wouldn't the big labs?

That is exactly the document I've been thinking about while writing top comment :) . Oh, our "cost of revenue" is smaller than the revenue, we are so profitable guys! If we just don't count our marketing expenses, our administrative expenses, and our unspecified losses from operations to the tune of x3 times higher than our revenue. But if we don't count them we are totally profitable guys! If we will just stop all RnD and also stop paying our salaries, support and marketing departments (and also cut 20 billion of operational losses too, whatever that is) we will earn soo soo much monies guys!

Re: Why current LLM costs are not sustainable

#186

Post by Eddy_Viscosity2 »

oarsinsync wrote: post #113
LUmBULtERA wrote: post #74

Earlier quoted context omitted.

>3. We're massively overusing SOTA models. As long as you're on a subsidized subscription, you can use Claude Opus 4.8 high to write blog article meta descriptions. If you paid by token, you wouldn't do that. This idea that the subscriptions are subsidized is repeated over and over, but I've never seen any proof of this. It seems to be entirely based on the inferred API cost the subscription usage could give you, but…

> This idea that the subscriptions are subsidized is repeated over and over, but I've never seen any proof of this. It seems to be entirely based on the inferred API cost the subscription usage could give you, but there are a lot of assumptions needed for that to follow. My claude code environment shows me cost per token used in that session, according to API costs. It regularly exceeds $200. I pay $200 a month for m…

You're being shown the price, not the cost.

Re: Why current LLM costs are not sustainable

#187

Post by hparadiz »

piva00 wrote: post #181
hparadiz wrote: post #151

Earlier quoted context omitted.

[flagged]

You can save it to your favourites, no need to comment at all if it's not going to add to the conversation.

Okay here is my adding to the conversation:

The current discourse about LLMs in coding especially is based on the cheapest type of inference: text. This technology was designed for images which is a much more computationally expensive task than text. If it's already profitable to use this technology for multimedia like images and videos then using it on a text based inference for code is less then 1% as computationally expensive. Furthermore in the aggregate over time the computational expensive of text based inference precipitates negatively. In other words using it to write code will inevitably become a throwaway computational task like decompressing a jpeg. And yes decompressing jpegs would lag your 386 in the early 90s.

Re: Why current LLM costs are not sustainable

#189

Post by _flux »

LUmBULtERA wrote: post #141
_flux wrote: post #125

Earlier quoted context omitted.

So they do not train models, and in addition their models are expected to be smaller than SOTA models, although we cannot know for sure by how much. So what's the price difference, 3000x?

My comment is about your statement "serving these tokens without paying for training is already expensive"... One thing we do know from OpenAI's leaked financial document is that they are already profitable on inference, though that data is not broken down by cost and revenue of API vs. subscription. One important factor is that subscription inference can be optimized in ways to reduce cost (e.g., usage limits, batch…

So I let ChatGPT do the legwork for me, but it does seem the price difference between inference for GPT-5.5 and open-weight frontier models like DeepSeek V4 Pro and Kimi K2.6, which both are smaller models and thus cheaper to run inference on, is only 8x or so.

Sources https://openai.com/business/pricing/#api says for GPT-5.5:

  Input:$5.00 / 1M tokens Cached input:$0.50 / 1M tokens Output:$30.00 / 1M tokens
and for https://docs.fireworks.ai/serverless/pricing DeepSeek V4 Pro:

  Input: $1.74 / 1M tokens Cached input: $0.145 / 1M tokens Output: $3.48 /
Ratios are: 2.8, 3.4, 8.6

So as these numbers seem reasonably comparable to SOTA, and the SOTA vendors have additional overhead, then I think it is fair to deem that the alternative explanation offered here is not the explanation:

> Why do you think that subscriptions are subsidized and not that enterprise tokens are sold at 3000% margin?

As it does seem like the GPT-5.5 API tokens do not have significant margin based on the overhead-free companies selling inference for smaller models at prices of the same scale, I think we can believe that the subscriptions must be heavily subsidized.

It should be noted though that DeepSeek itself sells this even cheaper, but they may also be in it for the getting market share.

Re: Why current LLM costs are not sustainable

#190

Post by sharpshadow »

kamaal wrote: post #167
sharpshadow wrote: post #160

Earlier quoted context omitted.

Deepseek will be sanctioned and therefore no provider will offer it anymore. Only way to use it then will be private but even that will be forbidden if it gets classified as threat to national security.

>>Deepseek will be sanctioned and therefore no provider will offer it anymore. That is possible inside the US. How do you do it all over the world? You have to convince every country in the world not use frontier models? Even worse how do you convince all the countries to not build their own models?

You are correct that it’s inside the US and it doesn’t apply to the whole world. But it will apply to the western world and much further which trades in dollar because else secondary sanctions apply. The shocking number is roughly 175-180 countries of 193 follow along US sanctions, that’s 90%. Unlike in the US the citizens of those countries will not be prosecuted for circumventing the sanctions but assets like bank accounts and wallets may be seized and Visa issues may arise. With the already ongoing legislation against foreign AI players and the recent national security threat assessment it strongly looks like just a matter of time until they implement the mechanics to sanction any disfavored AI. This will flush most users back to US companies. With the limited self hosting options and insanely strong export limits for datacenters it’s already established that countries can’t build their own models with the same power.
Post reply on HN