Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why current LLM costs are not sustainable

aditya.patadia.org

Post reply on HN
Print view View original

71–80 of 216 posts

Re: Why current LLM costs are not sustainable

#71

Post by xienze »

dofm wrote: post #51
walrus01 wrote: post #3

I have already seen a number of people doing the math on what it would take for hardware to self host a Q8XL quantization of GLM5.2 shared between N numbers of people. There's additional advantages that everything you query, all of your context cache and everything it outputs stays private and can't be arbitrarily turned off by external interference. Personally I think it would be a fairly good bet that something wit…

Back in the earlier days of the internet, when "dedicated servers" were a competitive advantage, hobbyists and small dev shops definitely shared dedicated hardware. So you could see small LLM co-operatives working out, yeah. But my thinking is that this four-to-five-year scenario just won't come to fruition, because the whole concept of needing to run these massive, massive models will slightly more likely be rendere…

> So you could see small LLM co-operatives working out, yeah.

Only on a pay-per-token basis, I think. Unless it's a very tight-knit circle of folks. Fixed monthly subscription costs I doubt would work in that model. Because you'll get the inevitable: someone pegging the service 24/7 because it's "unlimited" while everyone else suffers.

Re: Why current LLM costs are not sustainable

#72

Post by byzantinegene »

arbayi wrote: post #33

i think we have the causation backwards here. llms aren't expensive because they have to be — they're expensive because we keep reaching for the expensive model instead of putting any effort into making the cheap one good enough. a surprisingly large fraction of production workloads can be handled by smaller models with the right scaffolding. it's often easier to switch to a larger model than to engineer those pieces…

Unfortunately, the economics of what you suggest do not justify the trillion dollar valuations of OpenAI and Anthropic.

Re: Why current LLM costs are not sustainable

#73

Post by mschild »

yturijea wrote: post #68

I am using perhaps 15% of usage count on Claude with just the normal subscription. And I do full time software engineering and would say I use quite a lot of AI input on thoughts, designs and code drafts. So how these companies and people manage to use these absurd amount of tokens is a mystery to me. It feels like this are just running huge amount of non-vetted data to the LLM's and or running loops against the LLM'…

> So how these companies and people manage to use these absurd amount of tokens is a mystery to me.

Fire and forget. They run multiple agents in parallel 24/7. AI isn't just a rubber ducky for them, its their main (only) tool at that point.

Re: Why current LLM costs are not sustainable

#74

Post by LUmBULtERA »

FinnLobsien wrote: post #35

The problem space has a few aspects: 1. We're still in the "$5 airport Uber" era of LLMs. They're heavily subsidized, and everyone still complains about costs. 2. There hasn't been a real incentive to work on cost optimization for data centers and the hardware they contain. When/if price hikes happen and send people scrambling to use other models or drastically reduce AI usage, this will suddenly need to happen. 3. W…

>3. We're massively overusing SOTA models. As long as you're on a subsidized subscription, you can use Claude Opus 4.8 high to write blog article meta descriptions. If you paid by token, you wouldn't do that.

This idea that the subscriptions are subsidized is repeated over and over, but I've never seen any proof of this. It seems to be entirely based on the inferred API cost the subscription usage could give you, but there are a lot of assumptions needed for that to follow.

Re: Why current LLM costs are not sustainable

#76

Post by _pdp_ »

chiply314 wrote: post #55
_pdp_ wrote: post #45

Prices will go down one way or another. That is of course unless the market gets cornered by restricting model use, restricting supply of essential hardware components or raw materials to make this hardware, etc. In terms of running the model locally vs a service provider, that will be down to convenience more than anything else for the same reason why not everyone is hosting their own website at home on their own bo…

Token prices will go down for sure, but i watched a video interview on yt from cloudflare ceo and apparently the internet traffic of agentics increased and took over human. If we continue this year with a2a, agentic layer and co, there is probably a huge bulk coming up with a lot more agents running a lot longer and talking to each other to solve issues which will increase token usage significanlty.

My thinking is the same. I believe that AI will be the predominant forms of "intelligence" online probably taking as much as 90-99% of all traffic. It is not hard to see where this is going.

The price for tokens will become a proxy of consumed energy in my mind, i.e. tps will be something like kWh almost directly correlated in terms of cost.

Re: Why current LLM costs are not sustainable

#78

Post by dofm »

xienze wrote: post #71
dofm wrote: post #51

Earlier quoted context omitted.

Back in the earlier days of the internet, when "dedicated servers" were a competitive advantage, hobbyists and small dev shops definitely shared dedicated hardware. So you could see small LLM co-operatives working out, yeah. But my thinking is that this four-to-five-year scenario just won't come to fruition, because the whole concept of needing to run these massive, massive models will slightly more likely be rendere…

> So you could see small LLM co-operatives working out, yeah. Only on a pay-per-token basis, I think. Unless it's a very tight-knit circle of folks. Fixed monthly subscription costs I doubt would work in that model. Because you'll get the inevitable: someone pegging the service 24/7 because it's "unlimited" while everyone else suffers.

Well, many of us who shared hardware also ran monitoring to make sure the share was fair; there used to be a whole industry for that sort of quota stuff.

You can presumably hard-limit LLMs the same way — total, burst quotas etc.

(Suddenly getting a very fun flashback to the environment in which someone first explained Markov chains to me — MediaMOO. A text-based chat environment with configurable limits on the number of CPU "ticks" you were allowed in order to do things)

Re: Why current LLM costs are not sustainable

#79

Post by jstummbillig »

FinnLobsien wrote: post #35

The problem space has a few aspects: 1. We're still in the "$5 airport Uber" era of LLMs. They're heavily subsidized, and everyone still complains about costs. 2. There hasn't been a real incentive to work on cost optimization for data centers and the hardware they contain. When/if price hikes happen and send people scrambling to use other models or drastically reduce AI usage, this will suddenly need to happen. 3. W…

> We're still in the "$5 airport Uber" era of LLMs. They're heavily subsidized, and everyone still complains about costs.

Inference is not exactly cheap. Based on what do you think this is "heavily subsidized" still? What would to token cost have to be, with current models, for it to not be that? What do you know that has you make such a claim?

Re: Why current LLM costs are not sustainable

#80

Post by erfgh »

LUmBULtERA wrote: post #74
FinnLobsien wrote: post #35

The problem space has a few aspects: 1. We're still in the "$5 airport Uber" era of LLMs. They're heavily subsidized, and everyone still complains about costs. 2. There hasn't been a real incentive to work on cost optimization for data centers and the hardware they contain. When/if price hikes happen and send people scrambling to use other models or drastically reduce AI usage, this will suddenly need to happen. 3. W…

>3. We're massively overusing SOTA models. As long as you're on a subsidized subscription, you can use Claude Opus 4.8 high to write blog article meta descriptions. If you paid by token, you wouldn't do that. This idea that the subscriptions are subsidized is repeated over and over, but I've never seen any proof of this. It seems to be entirely based on the inferred API cost the subscription usage could give you, but…

They are subsidized by the huge losses incurred by the AI companies.
Post reply on HN