Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

211–220 of 319 posts

Re: LLMs are cheap

#211

Post by bustadjustme »

Sorry if I missed it, but how is a single token output from an LLM comparable to a search result from an engine? The author here compares 1k tokens (as an estimate for an average LLM single query response) to 1k web search queries. How is this not a factor of 1000 error?

> To compare a midrange pair on quality, the Bing Search vs. a Gemini 2.5 Flash comparison shows the LLM being 1/25th the price.

That is, 40x the price _per query_ on average (which is the unit of user interaction). LLMs with web-search will only multiply this value, as several queries are made behind the scenes for each user-query.

EDIT: thanks, zahlman, he does quote LLM prices in 1M tokens, or 1k user-queries, so the above concern is mistaken!

Re: LLMs are cheap

#212

Post by johnisgood »

fastball wrote: post #143
johnisgood wrote: post #53

Earlier quoted context omitted.

How about indirect? At any rate, something is going on, because our summers are more and more hotter, and there are no snow during our winters. We are all noticing it but it gets shrugged off as "misremembering". I am not contributing it to running LLMs alone, however, but climate change seems real enough to me, I experience it. It is barely July and I am dying! We used to have more tolerable weather around this time…

Yes, but what does climate change have to do specifically with LLMs? How are they different from any other use of energy? As far as I can tell they are better than most uses, given that (as software) they run entirely with electricity, which of course can be generated with near-zero CO2 emissions. Given that, this interjection about climate change seems like a complete non-sequitur to the topic at hand.

I do not think it has anything to do with LLMs. That is probably the least of our issues.

Re: LLMs are cheap

#214

Post by agentultra »

dale_glass wrote: post #149
agentultra wrote: post #110

Earlier quoted context omitted.

Most search engines will parse the query sentence much more intelligently than that. It's not literally matching every word and hasn't for decades. I just tried a handful of popular search engines, they all return the appropriate responses and links.

They're not that literal anymore of course, but they still don't compare to an LLM. In the end it's still mostly searching for key words even if with a few tweaks here and there, and the ability to answer vague questions mostly works by finding forums and Reddit posts where people ask that specific question and hopefully get an answer. When you're asking a standard question like the capital of whatever, that works gr…

> they still don't compare to an LLM

Of course they don't.

One is a program for seaching a corpus of data for items relevant to a query.

The other generates items from a corpus of data.

Re: LLMs are cheap

#215

Post by ahtihn »

rco8786 wrote: post #170
JimDabell wrote: post #125

Earlier quoted context omitted.

OpenAI spends less on training than inference, so the worst case scenario is less than double the cost after factoring in training. Inference is still cheap.

Inference is cheap. Training is cheaper. Then where's all the money going? OpenAI is reporting heavy losses, but you're saying the unit economics of inference are all good. What are they spending money on?

Salaries?

Re: LLMs are cheap

#216

Post by Palmik »

noodletheworld wrote: post #133
Palmik wrote: post #124

Earlier quoted context omitted.

> API that is likely a loss-leader to grab market share (hosted LLM cloud models). I don't think so, not anymore. If you look at API providers that host open-source models, you will see that they have very healthy margin between their API cost and inference hardware cost (this is, of course, not the only cost) [1]. And that does not take into account any proprietary inference optimizations they have. As for closed-mo…

I don’t completely disagree, but “assertion one” [1] [1] ~ you can obviously verify this yourself by doing it yourself and seeing how expensive it is. …is an enormously weak argument. You suppose. You guess. We guess. Let’s be honest, you can just stop at: > I don’t think so. Fair. I don’t either; but that’s about all we can really get at the moment afaik.

No, the point of [1] is that this is not some "secret knowledge". My response is based on running models in production and comparing my costs with the costs I would pay to API providers running the same models.

Re: LLMs are cheap

#217

Post by lazide »

bee_rider wrote: post #128
leeoniya wrote: post #112

Earlier quoted context omitted.

commercial motivatation needs to show eventual profit to be sustainable, while political does not. though at the outset (pre-profit / private) it's hard to say there's much difference.

> though at the outset (pre-profit / private) it's hard to say there's much difference. I think this is the tough part, we’re at the outset still. Also, a political investment could could be sustainable, in the sense that China might decide they are fine running Deepseek at a loss indefinitely, if that’s what’s going on (hypothetically. Actually I have never seen any evidence to suggest Deepseek is subsidized, althou…

Also, solar panel dumping as a quite successful example (on many, many fronts).

Re: LLMs are cheap

#218

Post by jakobnissen »

bustadjustme wrote: post #211

Sorry if I missed it, but how is a single token output from an LLM comparable to a search result from an engine? The author here compares 1k tokens (as an estimate for an average LLM single query response) to 1k web search queries . How is this not a factor of 1000 error? > To compare a midrange pair on quality, the Bing Search vs. a Gemini 2.5 Flash comparison shows the LLM being 1/25th the price. That is, 40x the p…

Gemini 2.0 Flash is listed at 0.4 USD / 1M tokens. Bing search API is 15 USD / 1k queries. So the LLM is indeed 37 times cheaper for a 1000 token query.

Re: LLMs are cheap

#219

Post by lazide »

otterley wrote: post #138
ddp26 wrote: post #134

Earlier quoted context omitted.

I analyzed OpenAI API profitability in summer 2024 and found inference for gpt-4 class models likely pretty profitable, ~50% gross margins (ignoring capex for training models): https://futuresearch.ai/openai-api-profit

That’s a little like saying you can compute the profitability of the energy market by looking only at the margins of gas stations. You can’t exclude all the outlays on actually acquiring the product to sell.

Sure - but is there any doubt in that example that gas stations are making a profit?

And unlike gasoline, once models are trained there is no significant ongoing production cost.

Re: LLMs are cheap

#220

Post by qeternity »

bee_rider wrote: post #186
JimDabell wrote: post #126

Earlier quoted context omitted.

> The nodes used for training are rented, so that’s opex, right? It’s capex. They are putting money in, and getting an asset out (the weights). > The models are in some sense consumable? Assets depreciate.

Obsolete software don’t depreciate like obsolete hardware. If an LLM company has trained a truly better model, they can simply make as many copies of their own model as they want. Thus, if the new model is truly better in every way, the old one is completely valueless to them (of course there might be some tradeoffs which mean older models can stick around because they are, say, smaller… but, ultimately they will be…

> the old one is completely valueless to them

This is of course untrue for the same reason that people are still running Windows 2000.

Post reply on HN