Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why current LLM costs are not sustainable

aditya.patadia.org

Post reply on HN
Print view View original

191–200 of 216 posts

Re: Why current LLM costs are not sustainable

#191

Post by auggierose »

yturijea wrote: post #179
auggierose wrote: post #83

Earlier quoted context omitted.

What is a "normal" subscription? Are you using Claude Code, or just Claude??

Just Claude, I have seen the weird hallucinations these LLM's make, yes that also means opus, fable etc. so I don't trust it to just run its own clause. Yes that also means I get to inspect and confirm every step of the way, to ensure the design is followed, we are not making unneccesary changes, we have thought about edge cases, testing etc. And I also keep an understanding of what is produced, because I will manual…

I get where you are coming from (it's me, last year), and that's what I thought.

But if you use Claude Code or Codex, you will blow through your pro plan quickly. If you don't use them, you are not really using AI. I know how that sounds, but that is how it is.

These models are smart now. Really smart. Yes, they hallucinate, but usually not without reason. I am having long discussions with these models before generating code, and generate markdown from them. These are then the basis for the generated code. I am trying to give the model as much background as possible. I read the generated markdown: if there is something that feels off, like I don't really know what it means, then you need to fix that first, by discussing it with the model. Often, these are real problems in how I was understanding something, the model wasn't really getting it, and just made something up that it hoped would kinda work.

And I prefer Codex over Claude Code (prior to Fable, Fable is something else!), it behaves more like a helpful PhD-level colleague and just feels sharper. Claude Code sounds a bit like a mix between an HR person and a therapist that is on vacation too often.

I am still looking at code, but only if something came up during high-level discussions with the model that I want to pin down exactly. Otherwise I just talk about the high-level intention of the code, usually not looking at it.

What REALLY helps is coming up with the right theoretical frameworks for your work, with practical implementations that the model can use, and that allow some kind of verification. Let's say you want to parse something. For a one-off the model is great at generating "hand-rolled" parsing code, but for something disciplined, giving the model a way to generate context-free grammars and giving it a way to check them for determinism gives great results.

Re: Why current LLM costs are not sustainable

#192

Post by piva00 »

hparadiz wrote: post #187
piva00 wrote: post #181

Earlier quoted context omitted.

You can save it to your favourites, no need to comment at all if it's not going to add to the conversation.

Okay here is my adding to the conversation: The current discourse about LLMs in coding especially is based on the cheapest type of inference: text. This technology was designed for images which is a much more computationally expensive task than text. If it's already profitable to use this technology for multimedia like images and videos then using it on a text based inference for code is less then 1% as computational…

What?

LLMs were designed for text, it's in their name "large language model". Only with specialised encoders like vision transformers they were able to process images as well but you're absolutely wrong about the original design intent.

In the end you just added misinformation, just save the comment to your favourites and set a reminder to check it again in a few years like you wanted.

Re: Why current LLM costs are not sustainable

#193

Post by hparadiz »

piva00 wrote: post #192
hparadiz wrote: post #187

Earlier quoted context omitted.

Okay here is my adding to the conversation: The current discourse about LLMs in coding especially is based on the cheapest type of inference: text. This technology was designed for images which is a much more computationally expensive task than text. If it's already profitable to use this technology for multimedia like images and videos then using it on a text based inference for code is less then 1% as computational…

What? LLMs were designed for text, it's in their name "large language model". Only with specialised encoders like vision transformers they were able to process images as well but you're absolutely wrong about the original design intent. In the end you just added misinformation, just save the comment to your favourites and set a reminder to check it again in a few years like you wanted.

The first technological breakthroughs were with face and red eye detection in 2003. Then object detection between 2008-2012. Text models didn't become useful until about 2016. Please watch the first course of Dr Fei Fei Li's lectures on the subject.

Re: Why current LLM costs are not sustainable

#194

Post by szatkus »

kamaal wrote: post #162
yturijea wrote: post #68

I am using perhaps 15% of usage count on Claude with just the normal subscription. And I do full time software engineering and would say I use quite a lot of AI input on thoughts, designs and code drafts. So how these companies and people manage to use these absurd amount of tokens is a mystery to me. It feels like this are just running huge amount of non-vetted data to the LLM's and or running loops against the LLM'…

>>So how these companies and people manage to use these absurd amount of tokens is a mystery to me. Absolutely! I know some colleagues who are routinely spending thousands of dollars worth of tokens, I can't see to even max out the subscription limits even if Im working all the time. Curiously enough their output is lower too.

Agents could work for a long time and burn a lot of tokens if you give them a task that is too hard for them. After enough time, if they don't give up, the slot machine could spit out a working solution.

Personally I find it faster to figure out the hard parts by myself and then give a few smaller tasks to Claude.

Re: Why current LLM costs are not sustainable

#195

Post by LUmBULtERA »

Yizahi wrote: post #185
usef- wrote: post #168

Earlier quoted context omitted.

OpenAI's leaked documents also said OpenAI was profitable on inference. The small resellers of open models have nowhere near the resources to optimise their models or inference and yet usually have a lower cost, why wouldn't the big labs?

That is exactly the document I've been thinking about while writing top comment :) . Oh, our "cost of revenue" is smaller than the revenue, we are so profitable guys! If we just don't count our marketing expenses, our administrative expenses, and our unspecified losses from operations to the tune of x3 times higher than our revenue. But if we don't count them we are totally profitable guys! If we will just stop all R…

Fair to include A&G, but their marketing and training cost is about generating future growth. People say that the subscriptions are highly subsidized as if its fact -- it's certainly not a well-established fact. These people simply don't know the truth one way or another, but portray their not well supported opinion as fact.

Re: Why current LLM costs are not sustainable

#196

Post by jijijijij »

LUmBULtERA wrote: post #182
jijijijij wrote: post #172

Earlier quoted context omitted.

Sorry, I don't understand what you are trying to say.

The craftsman, who may otherwise be profitable, also has investment costs that cause them to show a loss for some time.

"Otherwise". If the craftsman revenue isn't enough to recover the investment expenses, the business is operating at loss. But that's beside the point, because research investments are not the issue at hand.

As said before: Interference costs are not the only operational costs. Same as electricity costs for the craftsman. Running a power drill is not the the whole expense to consider. The craftsman has to eat, AI company's employees have to eat. The craftsman has to learn about new building standards, the AI company has to train their models because no one wants to use a product stuck in time (that's not "research", just maintenance). If not even interference was recovered in revenue, nobody would even start to argue about sustainability.

I can't debate this further, because HN is rate limiting my account for dissenting opinions in the past.

Re: Why current LLM costs are not sustainable

#197

Post by brandensilva »

LUmBULtERA wrote: post #146
oarsinsync wrote: post #113

Earlier quoted context omitted.

> This idea that the subscriptions are subsidized is repeated over and over, but I've never seen any proof of this. It seems to be entirely based on the inferred API cost the subscription usage could give you, but there are a lot of assumptions needed for that to follow. My claude code environment shows me cost per token used in that session, according to API costs. It regularly exceeds $200. I pay $200 a month for m…

The API inference cost to customers is not the actual cost of providing inference, and the cost of providing API inference need not be the cost of providing subscriber inference.

This is correct, they are subsidized but it's the training cost that costs the most with a majority of people hitting cache for most queries for inference.

Re: Why current LLM costs are not sustainable

#198

Post by square_usual »

Aldipower wrote: post #96
KronisLV wrote: post #6

> To give an example, just doing Typescript type fixes with this model across 50 files cost me $54 this afternoon. If you can use a subscription with any of the SOTA models, do that. Instead of around 4k EUR in token costs, my Opus usage costs me 108 EUR (with taxes) per month with their Max 5x plan. It's the same with OpenAI, those are heavily subsidized. It doesn't make sense to pay per-token, unless you must. > Wh…

> If you can use a subscription with any of the SOTA models, do that. Those subscriptions plans are for private use only! If you are running a business you are not allowed to use them actually. Anyway..

You can use them in the team plan, but not the enteprise plan.

Re: Why current LLM costs are not sustainable

#199

Post by FridgeSeal »

akie wrote: post #11

I am convinced that the combination of capable open weight models and specialized hardware will mean that Apple (and other hardware providers) will start shipping computers with built-in, hardwired, "LLM-on-a-chip" cards that are capable enough to meet 90% of your AI needs. I really believe that in the near-term future we will run our LLMs in hardware, not in software. Hardwire a capable model into a device the size…

I’d put money on Apple buying/acqui-hiring the Talaas people to further this.

Re: Why current LLM costs are not sustainable

#200

Post by jijijijij »

usef- wrote: post #180
jijijijij wrote: post #171

Earlier quoted context omitted.

> Just because there's time between big releases doesn't mean progress stopped. No, but progress not stopping doesn't mean it's not plateauing. I believe 'plateauing' is understood as the process of approaching a plateau, not being stuck on a plateau already. So, the question is about the rate of progress, not its existence.

I guess we draw a different line then. This year has been full of a lot of great releases so far. Most normal people didn't even use Agents before January. It does not at all feel slower than previous years. HN commenters have been saying that LLMs plateaued ever since the first ChatGPT release. 6 months ago: > LLMs are amazing, but they have reached a plateau. https://news.ycombinator.com/item?id=46109534 1 year ago…

It doesn't matter where we draw the line, or what someone else said at some point. I merely pointed out your argument wasn't a sound rebuttal. Criticizing subjective experiences to assess the situation is, but that cuts both ways.

> HN commenters have been saying that LLMs plateaued ever since the first ChatGPT release.

Your earliest example was 2 years after release, when LLMs were already widely used and there is literally a source to support the claim. Now, you need to show research efforts, time and resource investments, ... are producing proportional results, disproving diminishing returns. If there are diminishing returns, LLMs are plateauing.

Also, manual capability extensions, or use/edge case adaptations, which may improve subjective usability are not exactly advances in AI as technology. LLMs still hallucinate. LLms still fundamentally struggle with certain classes of problems (e.g. counting), but you increasingly need to come up with different problem dress-ups because of targeted interventions to manufacture hype and a limited supply of test cases. Can you make the case AI got actually more intelligent, fundamentally? That is, not an increase in case specific usability, but a decrease in fundamental limitations. And is this proportional to improvement efforts?

Post reply on HN