Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why current LLM costs are not sustainable

aditya.patadia.org

Post reply on HN
Print view View original

41–50 of 216 posts

Re: Why current LLM costs are not sustainable

#41

Post by user43928 »

KronisLV wrote: post #6

> To give an example, just doing Typescript type fixes with this model across 50 files cost me $54 this afternoon. If you can use a subscription with any of the SOTA models, do that. Instead of around 4k EUR in token costs, my Opus usage costs me 108 EUR (with taxes) per month with their Max 5x plan. It's the same with OpenAI, those are heavily subsidized. It doesn't make sense to pay per-token, unless you must. > Wh…

Opus 4.8 High effort seems adequate for me currently, at API pricing, with a $200/month budget.

This is at work where I don't work on greenfield or parallelize feature development.

I cannot see the agent burning through $50 for one moderately sized TypeScript cleanup in my setup. This sounds like something that can be improved on OP's side.

There have been rumors about a potential Sonnet 5 model release in the near future, which hopefully tilts the cost/benefit ratio further in our favor.

Re: Why current LLM costs are not sustainable

#43

Post by simianwords »

The author understands well that Opensource is catching up but I think that the gap will remain constant - SOTA models will still be more performant.

The author mentions $54 in costs but the reality is that developers are paid around this much per hour.

What is likely to happen: LLM performance goes even higher and can do tasks that take humans days to accomplish. You then have to compare LLM cost with human cost - something the Author has forgotten in their analsys.

Re: Why current LLM costs are not sustainable

#44

Post by rvz »

This is no surprise at all and was very predictable.

The Chinese open weight models were always winning the AI race to zero where as the likes of Anthropic and OpenAI have no choice but to increase token costs.

Even Microsoft wants to use some of the Chinese models only realizing how expensive both the frontier models are. It turns out that Jevon's paradox does not exist in the US (it exists in China).

This "Tokenmaxxing" marketing stunt was a scam for the frontier models to raise even more money at unsustainable valuations.

Re: Why current LLM costs are not sustainable

#45

Post by _pdp_ »

Prices will go down one way or another. That is of course unless the market gets cornered by restricting model use, restricting supply of essential hardware components or raw materials to make this hardware, etc.

In terms of running the model locally vs a service provider, that will be down to convenience more than anything else for the same reason why not everyone is hosting their own website at home on their own box.

Re: Why current LLM costs are not sustainable

#46

Post by chiply314 »

lukebuehler wrote: post #5
chiply314 wrote: post #2

I think companies will fire 5-10% of people and convert them to token budget. I also believe that before any real companies are running these models locally, they will already have some kind of agentic layer. With the current frontier model lab progress, i do not see any real company which makes real money, running local models. Running local models is easy for me, for sure not that easy for any company. Your DC need…

I work with large enterprises that _only_ run critical workloads on locally hosted models. Think banks, insurance, etc--businesses that absolutely cannot leak any data. They also have CC and Codex, but their use is extremely restricted; anything of consequence runs on models running on GPU clusters in their own datacenter.

I work at large enterprise and they are happy paying Microsoft and AWS for model hosting.

But for sure there will be use cases of very critical data, but at the end the question will still be how big they are in comparision to the rest of the market.

These cricial workloads also have the cost issue, right? so will they reduce workforce to compensate for the budget?

Re: Why current LLM costs are not sustainable

#47

Post by FinnLobsien »

eru wrote: post #37
FinnLobsien wrote: post #35

The problem space has a few aspects: 1. We're still in the "$5 airport Uber" era of LLMs. They're heavily subsidized, and everyone still complains about costs. 2. There hasn't been a real incentive to work on cost optimization for data centers and the hardware they contain. When/if price hikes happen and send people scrambling to use other models or drastically reduce AI usage, this will suddenly need to happen. 3. W…

Mostly agreed, however I'm not sure about 3: I suspect it works like gym memberships, and the companies mostly make their money from people who don't use the subscriptions all that much.

I think the problem is that the companies mostly don't make money, period. They may have better unit economics on underused subscriptions, but I don't see a world in which OAI/Anthropic don't heavily tighten the screws in the future.

Right now it's silly to default to frontier models, but it won't bankrupt your company. I believe in the short-medium term future, we'll need to be more deliberate about model choices.

In the long-term, of course, tech costs tend to plummet. Is there a future where in 15 years, my Apple Watch locally runs an Opus 4.8-class model? Maybe. And that would obviate this whole discussion.

Re: Why current LLM costs are not sustainable

#48

Post by pjc50 »

juleiie wrote: post #9
arjunchint wrote: post #8

There is a wave of users switching over to DeepSeek Flash. There are Reddit threads of users sharing billion token spend for $20. If all of global spend on Anthropic/OpenAI/Gemini APIs just switches over to DeepSeek then easily we can decrease total AI spend by 10x

I am not sure if that is wise. It’s a hostile superpower after all

All AI is a hostile superpower. Might as well use the cheap one.

Re: Why current LLM costs are not sustainable

#49

Post by swiftcoder »

> To give an example, just doing Typescript type fixes with this model across 50 files cost me $54 this afternoon.

Who in hell would actually do this? That's a level of problem that any of the flash-class models can solve.

Hand that sort of thing to GPT-mini, Haiku, or DeepSeek Flash, and save the big guns for big architectural problems.

Re: Why current LLM costs are not sustainable

#50

Post by chiply314 »

netdevphoenix wrote: post #13
chiply314 wrote: post #2

I think companies will fire 5-10% of people and convert them to token budget. I also believe that before any real companies are running these models locally, they will already have some kind of agentic layer. With the current frontier model lab progress, i do not see any real company which makes real money, running local models. Running local models is easy for me, for sure not that easy for any company. Your DC need…

I am calling it now. LLM hosting is the new web hosting. You will have a market of hosting providers offering you access to LLM compatible hardware (the Hetzners of the LLM world) as well as virtualised LLM access (the Heroku of the LLM world). These will compete along pricing, ownership axes while frontier labs will compete mostly on performance, integration and ease of use (think Wordpress). That's the only way I c…

I work at a very big company and they just pay azure and aws to host claude and co for them.

So the hyperscalers already won for now probably.

At the end of the day, you send a lot of personal data to these endpoints. If you already host everything through microsoft already, LLM hosting is then a no brainer.

Post reply on HN