Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

131–140 of 319 posts

Re: LLMs are cheap

#131

Post by ozim »

xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

I think you can make an educated guess if you check local model performance, prices of energy and hardware and price of the subscriptions.

Best part is you can make perplexity research task out of it

Re: LLMs are cheap

#132

Post by JimDabell »

ceejayoz wrote: post #129
JimDabell wrote: post #115

Earlier quoted context omitted.

I just tried that search on Google. The first thing I saw was the AI summary. Underneath that was a third-party site. Underneath that was “People also ask” with five different questions. And then underneath that was the link to the American Airlines site. I followed the line to the official site. I was presented with a “We care about your privacy” consent screen, with four categories. The first category, “Strictly ne…

> Websites have been choosing shitty UX for decades at this point. The web is so polluted with crap and obstacles it’s ridiculous. Nobody seems to care any more. Now LLMs have come along that will just give you the info straight away without any fuss, so of course people are going to prefer them. Do you honestly believe LLMs aren't gonna get sponsored answers/ads and "helpful" UI elements that boost their profits?

I’m talking about today’s experience, not speculating about what might happen at some arbitrary point in the future.

The web has this shitty UX. LLMs do not have this shitty UX. I’m going to judge on what I can see and use.

Re: LLMs are cheap

#133

Post by noodletheworld »

Palmik wrote: post #124
xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

> API that is likely a loss-leader to grab market share (hosted LLM cloud models). I don't think so, not anymore. If you look at API providers that host open-source models, you will see that they have very healthy margin between their API cost and inference hardware cost (this is, of course, not the only cost) [1]. And that does not take into account any proprietary inference optimizations they have. As for closed-mo…

I don’t completely disagree, but “assertion one” [1]

[1] ~ you can obviously verify this yourself by doing it yourself and seeing how expensive it is.

…is an enormously weak argument.

You suppose. You guess. We guess.

Let’s be honest, you can just stop at:

> I don’t think so.

Fair. I don’t either; but that’s about all we can really get at the moment afaik.

Re: LLMs are cheap

#134

Post by ddp26 »

xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

I analyzed OpenAI API profitability in summer 2024 and found inference for gpt-4 class models likely pretty profitable, ~50% gross margins (ignoring capex for training models): https://futuresearch.ai/openai-api-profit

Re: LLMs are cheap

#135

Post by 827a »

Here's a question I don't know the answer to right off, but we're going to discover together in real time: Which is cheaper, one million queries to fetch the value of a secret from AWS Secrets Manager, or one million tokens from a modern LLM?

Storing a secret in secrets manager: $0.40

$0.05/10,000 API calls * 1,000,000 calls = $5

Total cost: $5.40.

Gemini 2.5 Flash: $0.15/million tokens.

Well, there you have it. Storing a secret value in AWS Secrets Manager is ~36 times more expensive per API call than querying an LLM!

Re: LLMs are cheap

#136

Post by jstummbillig »

xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

There is also a lot of different models at a lot of different price points (and LLMs are fairly hard to compare to begin with). In this theory of a likely loss-leader, must we assume that all of them, from all companies, are priced below cost...? If so, that seems like a fairly wild claim. What's Step 2 for all of these companies to get ahead of this, given how model development currently works?

I think the far more reasonable assumption is: It's profitable enough to not get super nervous about the existence of your company. You have to build very costly models and build insanely costly infrastructure. Running all of that at a loss without an obvious next step, because ALL of them are pricing to not even make money at inference, seems to require a lot of weird ideas about how companies are run.

Re: LLMs are cheap

#137

Post by ceejayoz »

JimDabell wrote: post #132
ceejayoz wrote: post #129

Earlier quoted context omitted.

> Websites have been choosing shitty UX for decades at this point. The web is so polluted with crap and obstacles it’s ridiculous. Nobody seems to care any more. Now LLMs have come along that will just give you the info straight away without any fuss, so of course people are going to prefer them. Do you honestly believe LLMs aren't gonna get sponsored answers/ads and "helpful" UI elements that boost their profits?

I’m talking about today’s experience, not speculating about what might happen at some arbitrary point in the future. The web has this shitty UX. LLMs do not have this shitty UX. I’m going to judge on what I can see and use.

> I’m talking about today’s experience…

In that case, get uBlock. The answer is in the first result, on the first screen, and the answer is even quoted in the short description from the site. (As a bonus, it also blocks the cookie consent popups on the AA site, if you like.)

The only thing getting in the way of the real, vetted, straight-from-the-source answer currently is the AI overview.

https://imgur.com/a/pRUGgRx

Re: LLMs are cheap

#138

Post by otterley »

ddp26 wrote: post #134
xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

I analyzed OpenAI API profitability in summer 2024 and found inference for gpt-4 class models likely pretty profitable, ~50% gross margins (ignoring capex for training models): https://futuresearch.ai/openai-api-profit

That’s a little like saying you can compute the profitability of the energy market by looking only at the margins of gas stations. You can’t exclude all the outlays on actually acquiring the product to sell.

Re: LLMs are cheap

#139

Post by WhyIsItAlwaysHN »

barrkel wrote: post #77
WhyIsItAlwaysHN wrote: post #65

There's something I don't get in this analysis. The queries for the LLM which were used to estimate costs don't make a lot of sense for LLMs. You would not ask an LLM to tell you the baggage size for a flight because there might be a rule added a week ago that changes this or the LLM might hallucinate the numbers. You would ask an LLM with web search included so it can find sources and ground the answer. This applies…

Oh contraire, I ask questions about recent things all the time, because the LLM will do a web search and read the web page - multiple pages - for me, and summarize it all. 4o will always do a web search for a pointedly current question, give references in the reply that can be checked, and if it didn't, you can tell it to search. o3 meanwhile will do many searches and look at the thing from multiple angles.

But that was my point, then you need to include the entire websites in the context and it won't be 506 tokens per question. It will be thousands

Re: LLMs are cheap

#140

Post by WhyIsItAlwaysHN »

brookst wrote: post #74
WhyIsItAlwaysHN wrote: post #65

There's something I don't get in this analysis. The queries for the LLM which were used to estimate costs don't make a lot of sense for LLMs. You would not ask an LLM to tell you the baggage size for a flight because there might be a rule added a week ago that changes this or the LLM might hallucinate the numbers. You would ask an LLM with web search included so it can find sources and ground the answer. This applies…

Just tried asking “what is the maximum carryon size for an American Airlines flight DFW-CDG” and it used a webs search, provided the correct answer, and provided links to both the airline and FAA sites. Why wouldn’t I use it like this?

What I was saying is that you wouldn't use a raw LLM (so 506 tokens to get an answer). You would use it with web search so you can get the links.

The LLM has to read the websites to answer you so that significantly increases the token count, since it has to include them in its input.

Post reply on HN