Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

121–130 of 319 posts

Re: LLMs are cheap

#121

Post by xpe »

paxys wrote: post #20

The entire comparison hinges on people only making simple factual searches ("what is the capital of USA") on both search engines and LLMs. I'm going to say that's far enough from the standard use case for both these sets of APIs to be entirely meaningless. - If I'm using a search engine, I want to search the web. Yes these engines are increasingly providing answers rather than just search results, but that's a UI/pro…

> The entire comparison hinges on people only making simple factual searches ... on both search engines and LLMs.

I disagree, but I can see why someone might say this, because the article's author writes:

> So let's compare LLMs to web search. I'm choosing search as the comparison since it's in the same vicinity and since it's something everyone uses and nobody pays for, not because I'm suggesting that ungrounded generative AI is a good substitute for search.

Still, the article's analysis of "is an LLM API subsidized or not?" does not _rely_ on a comparison with search engines. The fundamental analysis is straightforward: comparing {price versus cost} per unit (of something). The goal is figure out the marginal gain/loss per unit. For an LLM, the unit is often a token or an API call.

Summary: the comparison against search engine costs is not required to assess if an LLM APIs is subsidized or not.

Re: LLMs are cheap

#123

Post by bee_rider »

JimDabell wrote: post #101
xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

> you also don't have any evidence that they are profitable. Sure we do. Go to AWS or any other hosting provider and pay them for inference. You think AWS are going to subsidise your usage of somebody else’s models indefinitely? > All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achieve profitability yet. Yes, capex not opex. T…

> Yes, capex not opex. The cost of running inference is opex.

This seems sort of interesting, maybe (I don’t know business, though). I agree that the cost of running inference is part of the opex, but saying that doesn’t rule out putting other stuff in the opex bucket.

Currently these LLM companies train and models on rented Azure nodes in an attempt to stay at the head of the pack, to be well positioned for when LLMs become really useful in a “take many white collar jobs” sense, right?

So, is it really obvious what’s capex and what’s opex? In particular:

* The nodes used for training are rented, so that’s opex, right?

* The models are in some sense consumable? Or at least temporary. I mean, they aren’t cutting edge anymore after a year or so, and the open weights models are always sneaking up on them, so at least they aren’t a durable investment.

Re: LLMs are cheap

#124

Post by Palmik »

xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

> API that is likely a loss-leader to grab market share (hosted LLM cloud models).

I don't think so, not anymore.

If you look at API providers that host open-source models, you will see that they have very healthy margin between their API cost and inference hardware cost (this is, of course, not the only cost) [1]. And that does not take into account any proprietary inference optimizations they have.

As for closed-model API providers like OpenAI and Anthropic, you can make an educated guess based on the not-so-secret information about their model sizes. As far as I know, Anthropic has extremely good margins between API cost and inference hardware cost.

[1]: This is something you can verify yourself if you know what it costs to run those models in production at scale, hardware wise. Even assuming use of off-the-shelf software, they are doing well.

Re: LLMs are cheap

#125

Post by JimDabell »

rco8786 wrote: post #104
JimDabell wrote: post #101

Earlier quoted context omitted.

> you also don't have any evidence that they are profitable. Sure we do. Go to AWS or any other hosting provider and pay them for inference. You think AWS are going to subsidise your usage of somebody else’s models indefinitely? > All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achieve profitability yet. Yes, capex not opex. T…

AWS isn’t doing the training on those models.

OpenAI spends less on training than inference, so the worst case scenario is less than double the cost after factoring in training. Inference is still cheap.

Re: LLMs are cheap

#126

Post by JimDabell »

bee_rider wrote: post #123
JimDabell wrote: post #101

Earlier quoted context omitted.

> you also don't have any evidence that they are profitable. Sure we do. Go to AWS or any other hosting provider and pay them for inference. You think AWS are going to subsidise your usage of somebody else’s models indefinitely? > All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achieve profitability yet. Yes, capex not opex. T…

> Yes, capex not opex. The cost of running inference is opex. This seems sort of interesting, maybe (I don’t know business, though). I agree that the cost of running inference is part of the opex, but saying that doesn’t rule out putting other stuff in the opex bucket. Currently these LLM companies train and models on rented Azure nodes in an attempt to stay at the head of the pack, to be well positioned for when LLM…

> The nodes used for training are rented, so that’s opex, right?

It’s capex. They are putting money in, and getting an asset out (the weights).

> The models are in some sense consumable?

Assets depreciate.

Re: LLMs are cheap

#127

Post by wrsh07 »

andrew_lettuce wrote: post #51
dist-epoch wrote: post #39

Earlier quoted context omitted.

Stock price go up is another way a company is profitable. The amazon playbook for 10+ years.

Amazon made huge money as they captured more and more of the market and didn't return any of it. The company literally became worth more and more each year. Open AI continues to hemorrhage money.

I'm confused by this claim - OpenAI has pretty meaningful revenue.

If they monetized free users, they would have even better revenue. The linked post estimates eg $1 per user per month would flip them to profitable.

Re: LLMs are cheap

#128

Post by bee_rider »

leeoniya wrote: post #112
WithinReason wrote: post #99

Earlier quoted context omitted.

is that better or worse than commercially motivated?

commercial motivatation needs to show eventual profit to be sustainable, while political does not. though at the outset (pre-profit / private) it's hard to say there's much difference.

> though at the outset (pre-profit / private) it's hard to say there's much difference.

I think this is the tough part, we’re at the outset still.

Also, a political investment could could be sustainable, in the sense that China might decide they are fine running Deepseek at a loss indefinitely, if that’s what’s going on (hypothetically. Actually I have never seen any evidence to suggest Deepseek is subsidized, although I haven’t gone looking).

Re: LLMs are cheap

#129

Post by ceejayoz »

JimDabell wrote: post #115
adrian_b wrote: post #96

Earlier quoted context omitted.

I do not see which is the added benefit provided by the LLM in such cases, instead of doing yourself that web search, and for free.

I just tried that search on Google. The first thing I saw was the AI summary. Underneath that was a third-party site. Underneath that was “People also ask” with five different questions. And then underneath that was the link to the American Airlines site. I followed the line to the official site. I was presented with a “We care about your privacy” consent screen, with four categories. The first category, “Strictly ne…

> Websites have been choosing shitty UX for decades at this point. The web is so polluted with crap and obstacles it’s ridiculous. Nobody seems to care any more. Now LLMs have come along that will just give you the info straight away without any fuss, so of course people are going to prefer them.

Do you honestly believe LLMs aren't gonna get sponsored answers/ads and "helpful" UI elements that boost their profits?

Re: LLMs are cheap

#130

Post by bicepjai »

graemep wrote: post #97
fedeb95 wrote: post #46

I don't think LLMs are inherently "costly" or "cheap". This doesn't really matter. Gold is pricey, but its usages justify the cost. Will LLMs, as they are used and evangelized now, have a true positive return for those using it? In some domains it will, most probably not everywhere and not for everyone.

> Gold is pricey, but its usages justify the cost I understand the point, but gold is expensive because it is a traditionally agreed store of value, rather than because of its usage. Rhodium would be a better example.

I’m curious why gold usage justifies the cost ?
Post reply on HN