Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

301–310 of 319 posts

Re: LLMs are cheap

#301

Post by JimDabell »

rco8786 wrote: post #170
JimDabell wrote: post #125

Earlier quoted context omitted.

OpenAI spends less on training than inference, so the worst case scenario is less than double the cost after factoring in training. Inference is still cheap.

Inference is cheap. Training is cheaper. Then where's all the money going? OpenAI is reporting heavy losses, but you're saying the unit economics of inference are all good. What are they spending money on?

> Then where's all the money going?

They are giving vast amounts of inference away as part of their free tier to gain market share. I said inference is cheap, not that it is free. Giving away a large amount of a cheap product costs money.

> you're saying the unit economics of inference are all good

Free tiers do not contradict positive unit economics.

Re: LLMs are cheap

#302

Post by fedeb95 »

graemep wrote: post #97
fedeb95 wrote: post #46

I don't think LLMs are inherently "costly" or "cheap". This doesn't really matter. Gold is pricey, but its usages justify the cost. Will LLMs, as they are used and evangelized now, have a true positive return for those using it? In some domains it will, most probably not everywhere and not for everyone.

> Gold is pricey, but its usages justify the cost I understand the point, but gold is expensive because it is a traditionally agreed store of value, rather than because of its usage. Rhodium would be a better example.

traditions have value ;)

Re: LLMs are cheap

#303

Post by stavros »

dcre wrote: post #235
bee_rider wrote: post #229

Earlier quoted context omitted.

> This is of course untrue for the same reason that people are still running Windows 2000. What is the reason?

They’ve built processes around it and don’t feel like/can’t afford to/ don’t know to how change them.

I would agree if Windows 2000 had the exact same APIs as the next version, but it doesn't. LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities.

Re: LLMs are cheap

#304

Post by lazide »

otterley wrote: post #248
lazide wrote: post #219

Earlier quoted context omitted.

Sure - but is there any doubt in that example that gas stations are making a profit? And unlike gasoline, once models are trained there is no significant ongoing production cost.

Models aren't static. In order for them to remain relevant, they have to be constantly retrained with new data. Plus there's a model arms race going on and which will probably continue for the foreseeable future.

Fair point - though various distilling and retraining tricks do reduce the cost quite a bit. It’s not like everyone is doing all the work they had to do from scratch, every time.

Re: LLMs are cheap

#305

Post by n4r9 »

pr337h4m wrote: post #181
n4r9 wrote: post #73

Earlier quoted context omitted.

One of those arguments is: > there's not that much motive to gain API market share with unsustainably cheap prices. Any gains would be temporary, since there's no long-term lock-in, and better models are released weekly The goal may be not so much locking customers in, but outlasting other LLM providers whilst maintaining a good brand image. Once everyone starts seeing you as "the" LLM provider, costs can start going…

Uber and Lyft rely on network effects, which do not exist in any meaningful sense for LLM API providers.

Yeah, that's definitely a factor in the attempt to "undercut and outlast". I guess I have two defenses: firstly, network effects might not be crucial, it might be enough for there to be a small cost to changing provider; secondly, I imagine the providers are finding ways to use network effects to bolster adoption - e.g. "Find me a party date when all my friends are free, book the catering and message them with invites".

Re: LLMs are cheap

#306

Post by disgruntledphd2 »

famouswaffles wrote: post #183
disgruntledphd2 wrote: post #171

Earlier quoted context omitted.

I'll definitely buy that argument for OpenAI, but then why are Anthropic/XAI etc losing money? They don't have the same generous free tiers as OpenAI and yet they keep raising absurd amounts of money.

I mean I would still expect them to currently lose money ? Their tiers aren't as generous but they're still free free (i.e no revenue generation whatsoever, google search is free but they're still generating revenue per user via ads and such). I think the authors point isn't that inference is so cheap that they can be profitable without changing anything but that inference is now cheap enough for say ads (however tha…

> Their tiers aren't as generous but they're still free free

Certainly Claude's free tier is not generous, I basically ended up subscribing the first day I used it.

But, assuming that the losses are from the free tier, it's odd to me that Anthropic wouldn't be showing some kind of cash generation at this point.

Granted training is super expensive and they're hiring loads of people ahead of revenue, but if they were unit-cost profitable, one would have expected this to be leaked during one of (the many) funding rounds they've engaged in.

I'm mostly unconvinced by the author's analysis because of the above, but it's certainly food for thought to shift my prior that LLM modelling and service providing is a bad business.

Re: LLMs are cheap

#307

Post by maccard »

JimDabell wrote: post #300
maccard wrote: post #269

Earlier quoted context omitted.

> Go to AWS or any other hosting provider and pay them for inference. You think AWS are going to subsidise your usage of somebody else’s models indefinitely? Not indefinitely or for any undetermined scale, but AWS regularly subsidise up to 100k [0] in credits. It would not surprise me in the slightest if most. Inference is much cheaper than training and 100k in compute covers a decent amount of usage. Activate is tie…

I added “indefinitely” precisely because I wanted to rule out discussion of the free credits. Those are clearly a loss-leader to get people to choose AWS and isn’t relevant to how the true cost of inference.

The point is that all of these projects are only viable when salaries are VC funded and the opex of inference is close to 0. It’s easy to say that nobody will subsidise inference if you exclude the main subsidies

Re: LLMs are cheap

#308

Post by qeternity »

stavros wrote: post #303
dcre wrote: post #235

Earlier quoted context omitted.

They’ve built processes around it and don’t feel like/can’t afford to/ don’t know to how change them.

I would agree if Windows 2000 had the exact same APIs as the next version, but it doesn't. LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities.

> LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities.

I don't mean to be too pointed here, but it doesn't sound like you have built anything at scale with LLMs. They are absolutely not plug n play from a behavior perspective. Yes, there is API compatibility (text in, text out) but that is not what matters.

Even frontier SOTA models have their own quirks and specialties.

Re: LLMs are cheap

#309

Post by msgodel »

qeternity wrote: post #308
stavros wrote: post #303

Earlier quoted context omitted.

I would agree if Windows 2000 had the exact same APIs as the next version, but it doesn't. LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities.

> LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities. I don't mean to be too pointed here, but it doesn't sound like you have built anything at scale with LLMs. They are absolutely not plug n play from a behavior perspective. Yes, there is API compatibility (text in, text out) but that is not what matter…

When I've built things with them I've mostly considered the quirks defects.

Kind of like how httpds will have quirks but those aren't really a good thing and they're kind of plug and play.

Re: LLMs are cheap

#310

Post by stavros »

qeternity wrote: post #308
stavros wrote: post #303

Earlier quoted context omitted.

I would agree if Windows 2000 had the exact same APIs as the next version, but it doesn't. LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities.

> LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities. I don't mean to be too pointed here, but it doesn't sound like you have built anything at scale with LLMs. They are absolutely not plug n play from a behavior perspective. Yes, there is API compatibility (text in, text out) but that is not what matter…

What kind of quirks have you seen that the next model wasn't better at?
Post reply on HN