Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why current LLM costs are not sustainable

aditya.patadia.org

Post reply on HN
Print view View original

61–70 of 216 posts

Re: Why current LLM costs are not sustainable

#61

Post by veselin »

The more I think on the problem, the more I believe this will be solved with US interventions. And the interventions will increase inflation by a lot, so prices will not go down.

The other alternatives with LLMs becoming more expensive in an Uber-like move may not work due to a lot of competition. I also don't think usage will increase 10x. I don't always have coding tasks for an LLM despite it being good.

My reasons to believe so are outside of what interests HN community and I am neither endorsing this behavior, nor I think it is that simple. But US also has a huge debt that it must service. Wouldn't it be convenient if it was suddenly halved in actual value?

Re: Why current LLM costs are not sustainable

#64

Post by ludamad »

jeswin wrote: post #22

Would prefer not to offend the author, but I do believe this article has very little for the HN audience. No new insight, and no numbers or new information.

Is there any place with better curation? I notice quite a few articles summarizing the state of AI that feel redundant with one another

Re: Why current LLM costs are not sustainable

#65

Post by byzantinegene »

veselin wrote: post #61

The more I think on the problem, the more I believe this will be solved with US interventions. And the interventions will increase inflation by a lot, so prices will not go down. The other alternatives with LLMs becoming more expensive in an Uber-like move may not work due to a lot of competition. I also don't think usage will increase 10x. I don't always have coding tasks for an LLM despite it being good. My reasons…

unlikely scenario as the main mandate of the federal reserve is to keep inflation in check. inflation reaching such levels would also cause interest rates to rise astronomically, and this would make the debt harder to service

Re: Why current LLM costs are not sustainable

#66

Post by xienze »

simianwords wrote: post #43

The author understands well that Opensource is catching up but I think that the gap will remain constant - SOTA models will still be more performant. The author mentions $54 in costs but the reality is that developers are paid around this much per hour. What is likely to happen: LLM performance goes even higher and can do tasks that take humans days to accomplish. You then have to compare LLM cost with human cost - s…

> The author mentions $54 in costs but the reality is that developers are paid around this much per hour.

Sure, but imagine a situation where you've spent an hour going back and forth with the LLM trying to fix a problem and at the end of it you've only made minimal progress. Now you've spent an hour of your time AND $54 with little to show for it. It's a metric I don't think many people track: the cost of going in circles with an LLM for an extended period of time while burning tokens and still not resolving the problem.

Re: Why current LLM costs are not sustainable

#67

Post by simianwords »

xienze wrote: post #66
simianwords wrote: post #43

The author understands well that Opensource is catching up but I think that the gap will remain constant - SOTA models will still be more performant. The author mentions $54 in costs but the reality is that developers are paid around this much per hour. What is likely to happen: LLM performance goes even higher and can do tasks that take humans days to accomplish. You then have to compare LLM cost with human cost - s…

> The author mentions $54 in costs but the reality is that developers are paid around this much per hour. Sure, but imagine a situation where you've spent an hour going back and forth with the LLM trying to fix a problem and at the end of it you've only made minimal progress. Now you've spent an hour of your time AND $54 with little to show for it. It's a metric I don't think many people track: the cost of going in c…

That happens with humans too and for sure LLMs make it better not worse.

I know the number of times I tried to do something where the answer was simple but I took a few days to get there.

Re: Why current LLM costs are not sustainable

#68

Post by yturijea »

I am using perhaps 15% of usage count on Claude with just the normal subscription. And I do full time software engineering and would say I use quite a lot of AI input on thoughts, designs and code drafts.

So how these companies and people manage to use these absurd amount of tokens is a mystery to me. It feels like this are just running huge amount of non-vetted data to the LLM's and or running loops against the LLM's which only produce fractional results if not wasted results for insane cost.

So really it is the equivalent of just burning money, or heating your house in the winter while having all your windows open.

Re: Why current LLM costs are not sustainable

#69

Post by starchild3001 »

A few thoughts:

1. Chat, being 3 yr old, is a fairly mature and solved problem today. Top companies aren't even talking about it anymore! Gemma 31B does it amazingly well (for $0.4/1M token output). Practically every near-SoTA and SoTA model does simple "chat-like" QA amazingly well -- summarization, basic question answering, single- or few-step search.

2. Tasks -- or knowledge work on a computer -- are the new frontier. Computers have become competent only recently, and only for some of the tasks so far. I'd guess another 2-3 yr development cycle, after which "el cheapo" models will be virtually indistinguishable from SoTA.

As tasks are the new game in town, AI labs can still charge a premium for it. That premium has disappeared already for chat; most users cannot tell 99% correct answer from 95% correct answer; nor do they always wish for maximum accuracy.

3. What comes after Tasks? I think today's AI startups should figure that one out and solve it before everyone else.

Re: Why current LLM costs are not sustainable

#70

Post by Cthulhu_ »

eru wrote: post #37
FinnLobsien wrote: post #35

The problem space has a few aspects: 1. We're still in the "$5 airport Uber" era of LLMs. They're heavily subsidized, and everyone still complains about costs. 2. There hasn't been a real incentive to work on cost optimization for data centers and the hardware they contain. When/if price hikes happen and send people scrambling to use other models or drastically reduce AI usage, this will suddenly need to happen. 3. W…

Mostly agreed, however I'm not sure about 3: I suspect it works like gym memberships, and the companies mostly make their money from people who don't use the subscriptions all that much.

I'd say that is/was their long game, but it's still very much in hype phase so there's a lot of people intensively using these models, and I don't think it's anywhere near cost efficient right now. Maybe in the long run when people get bored with it, but on the other hand people are becoming dependent on it for everyday things.

We've already seen price hikes / token limits earlier this year, with suddenly some people running out of budget on the first day of the month. This will likely keep going for a while.

On the other hand, costs will drop too - open models and specialized hardware, as the article notes. The long question will be whether the companies will get a return on their invested billions. I don't think they will, not with the amount of competition they're facing, and I don't think any one company or model (series) has a monopoly yet. Popularity sure, but I'm confident a competitor may appear tomorrow and people will switch.

Post reply on HN