Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

291–300 of 319 posts

Re: LLMs are cheap

#291

Post by jackdeansmith »

cush wrote: post #287
jackdeansmith wrote: post #280

Earlier quoted context omitted.

Want to bet? I'll give you 5:1 odds that tokens from a model with some specific benchmark performance (we can sort out the specific benchmarks or basket of benchmarks if you want to bet) will be cheaper two years from now.

Sure. To clarify, I'm not asserting that in two years today's 4o will be 100x more expensive, but the sum of many core offerings from various companies will be. It won't be unheard of for people to spend $2k-$10k/yr between many AI services

I misunderstood your comment then, my assertion is that models which have the same capabilities as current models will be cheaper in the future. I have no doubts that models with more capabilities will be more expensive.

Re: LLMs are cheap

#293

Post by sireat »

FergusArgyll wrote: post #258
raydev wrote: post #233

Earlier quoted context omitted.

sama tweeted that the $200 tier was priced too low to cover costs a few months ago.

At that price level you run into serious adverse selection

Meaning someone paying $200 monthly is going to use it as much as possible to get their money's worth.

I slightly disagree.

My hypothesis would be that the distribution for $200 users would be bimodal.

That is there would be a one concentration of super heavy power users.

The second concentration would be of people who want the "best AI" but are not power users and feel that most expensive -> the best.

Their actual usage would be just like normal free tier of ChatGPT.

Re: LLMs are cheap

#294

Post by graemep »

bicepjai wrote: post #130
graemep wrote: post #97

Earlier quoted context omitted.

> Gold is pricey, but its usages justify the cost I understand the point, but gold is expensive because it is a traditionally agreed store of value, rather than because of its usage. Rhodium would be a better example.

I’m curious why gold usage justifies the cost ?

Just to be clear, I am sayings usage does not.

Re: LLMs are cheap

#295

Post by rolandog »

wredcoll wrote: post #277
raydev wrote: post #261

Earlier quoted context omitted.

You think "we're losing money on this subscription tier" is good PR for their investors?

"This car is priced so low we're practically giving it away!"

But are you owning something with a subscription in a market that has normalized hiking their prices?

There used to be contracts with service providers, and that — IIRC — usually shielded consumers from exorbitant increases.

Re: LLMs are cheap

#296

Post by TZubiri »

nitwit005 wrote: post #272
pr337h4m wrote: post #181

Earlier quoted context omitted.

Uber and Lyft rely on network effects, which do not exist in any meaningful sense for LLM API providers.

Brand is huge in every market. It's hard to get people to visit your website at all. People know about OpenAI, and look it up.

No network effect + already profitable.

Not at all like Uber, let it go

Re: LLMs are cheap

#297

Post by raxxorraxor »

raincole wrote: post #148
xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

> an API that is likely a loss-leader to grab market share (hosted LLM cloud models) Everyone just repeats this but I never buy it. There is literally a service that allows you to switch models and service providers seamlessly (openrouter). There is just no lock-in. It doesn't make any financial sense to "grab market share". If you sell something with UI, like ChatGPT (the web interface) or Cursor, sure. But selling…

Depends on the business case. LLM slowly creep into several workflows and here determinism becomes more important than the latest abilities in reasoning.

People start to let their LLM parse text content. Be that mails, chats or transcriptions, the models often need to formalize their output and switching models can become burdensome, while developers might switch models on a whim.

Doesn't mean you can capture a market by selling cheap though.

Re: LLMs are cheap

#298

Post by wkat4242 »

antman wrote: post #105
JimDabell wrote: post #101

Earlier quoted context omitted.

> you also don't have any evidence that they are profitable. Sure we do. Go to AWS or any other hosting provider and pay them for inference. You think AWS are going to subsidise your usage of somebody else’s models indefinitely? > All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achieve profitability yet. Yes, capex not opex. T…

No we don't, MS used their OpenAI position as a strategy to increase Azure adoption. I am surprised AWS didn't give ls for free

Microsoft's plans for openai are much bigger than just Azure. They have 30 products/services with copilot in the name already. And they're pushing them like crazy to end users.

Re: LLMs are cheap

#299

Post by wkat4242 »

chipsrafferty wrote: post #288
SEGyges wrote: post #142

Earlier quoted context omitted.

Every LLM provider caches their KV-cache, it's a publicly documented technique (go stuff that KV in redis after each request, basically) and a good engineering team could set it up in a month.

Are you saying if I ask a prompt "foo" and then a month later another user asks "foo" then it retrieves a cached value?

No, the key value cache is the context in a way the model can read it.

Re: LLMs are cheap

#300

Post by JimDabell »

maccard wrote: post #269
JimDabell wrote: post #101

Earlier quoted context omitted.

> you also don't have any evidence that they are profitable. Sure we do. Go to AWS or any other hosting provider and pay them for inference. You think AWS are going to subsidise your usage of somebody else’s models indefinitely? > All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achieve profitability yet. Yes, capex not opex. T…

> Go to AWS or any other hosting provider and pay them for inference. You think AWS are going to subsidise your usage of somebody else’s models indefinitely? Not indefinitely or for any undetermined scale, but AWS regularly subsidise up to 100k [0] in credits. It would not surprise me in the slightest if most. Inference is much cheaper than training and 100k in compute covers a decent amount of usage. Activate is tie…

I added “indefinitely” precisely because I wanted to rule out discussion of the free credits. Those are clearly a loss-leader to get people to choose AWS and isn’t relevant to how the true cost of inference.
Post reply on HN