Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why current LLM costs are not sustainable

aditya.patadia.org

Post reply on HN
Print view View original

51–60 of 216 posts

Re: Why current LLM costs are not sustainable

#51

Post by dofm »

walrus01 wrote: post #3

I have already seen a number of people doing the math on what it would take for hardware to self host a Q8XL quantization of GLM5.2 shared between N numbers of people. There's additional advantages that everything you query, all of your context cache and everything it outputs stays private and can't be arbitrarily turned off by external interference. Personally I think it would be a fairly good bet that something wit…

Back in the earlier days of the internet, when "dedicated servers" were a competitive advantage, hobbyists and small dev shops definitely shared dedicated hardware.

So you could see small LLM co-operatives working out, yeah.

But my thinking is that this four-to-five-year scenario just won't come to fruition, because the whole concept of needing to run these massive, massive models will slightly more likely be rendered moot by smaller models with better reasoning capacity, and possibly even in that timescale by hardware innovations.

One of the biggest problems I have with the whole "we won't be profitable until 2030" model is that 2030 is almost exactly as far into the future as the launch of ChatGPT is in the past, and in that time, models far more capable than that first ChatGPT have been made available to freely download and run on desktop hardware that existed before it launched, and the entire non-model surrounding functionality of that original ChatGPT plus many more functions is now not much more than a routine weekend coding project.

I don't know why the market would entertain the idea that no upset like that is possible in the same period of time again.

Re: Why current LLM costs are not sustainable

#52

Post by bflesch »

albertgoeswoof wrote: post #4

One thing missing here is the maturity of agent harnesses. I’m finding the free deepseek flash model in opencode can handle all of my simple tasks, because the harness is so good. Soon that will be a local model. And the reality is that other industries aren’t finding the use for LLMs as much as programmers are. Sure there are some benefits but you can’t fire your marketing department and replace it with AI

AI is google-in-a-box, and there will be dedicated hardware to run it locally like there was with the crypto ASICs.

I feel the only ones losing are the AI startups and Google. This is why they're trying to morph into a social-media like experience of simulated human interaction that can monetize a certain demographic of vulnerable people.

Re: Why current LLM costs are not sustainable

#53

Post by Mashimo »

pydry wrote: post #15
arjunchint wrote: post #8

There is a wave of users switching over to DeepSeek Flash. There are Reddit threads of users sharing billion token spend for $20. If all of global spend on Anthropic/OpenAI/Gemini APIs just switches over to DeepSeek then easily we can decrease total AI spend by 10x

Probably won't be too long before the government decides to block deepseek's website based on "security" concerns.

Who is "The government"?

China or your local one?

Re: Why current LLM costs are not sustainable

#54

Post by skerit »

exizt88 wrote: post #19

> We are seeing improvements with each model release these days but it’s clear that the improvements are getting smaller and smaller. This is obviously untrue, both with GPT-5.4, and Claude Fable as examples in the last 6 months.

> the improvements are getting smaller and smaller

The AI haters have been saying this for 2 years now.

Re: Why current LLM costs are not sustainable

#55

Post by chiply314 »

_pdp_ wrote: post #45

Prices will go down one way or another. That is of course unless the market gets cornered by restricting model use, restricting supply of essential hardware components or raw materials to make this hardware, etc. In terms of running the model locally vs a service provider, that will be down to convenience more than anything else for the same reason why not everyone is hosting their own website at home on their own bo…

Token prices will go down for sure, but i watched a video interview on yt from cloudflare ceo and apparently the internet traffic of agentics increased and took over human.

If we continue this year with a2a, agentic layer and co, there is probably a huge bulk coming up with a lot more agents running a lot longer and talking to each other to solve issues which will increase token usage significanlty.

Re: Why current LLM costs are not sustainable

#56

Post by iamacyborg »

eru wrote: post #37
FinnLobsien wrote: post #35

The problem space has a few aspects: 1. We're still in the "$5 airport Uber" era of LLMs. They're heavily subsidized, and everyone still complains about costs. 2. There hasn't been a real incentive to work on cost optimization for data centers and the hardware they contain. When/if price hikes happen and send people scrambling to use other models or drastically reduce AI usage, this will suddenly need to happen. 3. W…

Mostly agreed, however I'm not sure about 3: I suspect it works like gym memberships, and the companies mostly make their money from people who don't use the subscriptions all that much.

I follow a guy called Daniel McCarthy on LinkedIn who writes a lot on CLV and that seems to be his take. Even if theoretically you get way more than you pay with subscriptions, the vast majority of people are not power users.

https://danielminhmccarthy.com/

Re: Why current LLM costs are not sustainable

#57

Post by dofm »

iamacyborg wrote: post #56
eru wrote: post #37

Earlier quoted context omitted.

Mostly agreed, however I'm not sure about 3: I suspect it works like gym memberships, and the companies mostly make their money from people who don't use the subscriptions all that much.

I follow a guy called Daniel McCarthy on LinkedIn who writes a lot on CLV and that seems to be his take. Even if theoretically you get way more than you pay with subscriptions, the vast majority of people are not power users. https://danielminhmccarthy.com/

The vast majority of active users of ChatGPT could successfully use a model like Gemma 4 12B with agentic search if x86 hardware didn't make that so difficult.

Likely even the E4B, which is really both fun and impressive.

That is clearly a big component of Apple's bet, anyway.

Re: Why current LLM costs are not sustainable

#59

Post by raincole »

> To give an example, just doing Typescript type fixes with this model across 50 files cost me $54 this afternoon.

1. How much it costs in terms of programmers' salaries?

2. Can DeepSeek do this (I bet it can) and how much it costs?

The fact the author ever had the idea of using a SOTA to solve do this means LLMs are actually quite cheap.

Re: Why current LLM costs are not sustainable

#60

Post by chiply314 »

rubin55 wrote: post #27
exizt88 wrote: post #19

> We are seeing improvements with each model release these days but it’s clear that the improvements are getting smaller and smaller. This is obviously untrue, both with GPT-5.4, and Claude Fable as examples in the last 6 months.

I would struggle to ascertain the day-to-day difference between GPT-5.4 and GPT-5.5 tbh. Also, imho, Fable is highly hyped, I don't think it is dramatically better than Opus 4.8. Maybe my tasks and interaction with AI is relatively simple (i.e., lots of Rust programming, Linux system engineering stuff).

I haven't had enough time with fable, but I had to look back on how i worked with claude just 6 month ago to remind myself that it got a lot better.

Like i still used plan mode 6 month ago now I don't.

I would argue that with every model release we have a new learning phase.

Post reply on HN