Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

171–180 of 319 posts

Re: LLMs are cheap

#171

Post by disgruntledphd2 »

famouswaffles wrote: post #79
paxys wrote: post #20

The entire comparison hinges on people only making simple factual searches ("what is the capital of USA") on both search engines and LLMs. I'm going to say that's far enough from the standard use case for both these sets of APIs to be entirely meaningless. - If I'm using a search engine, I want to search the web. Yes these engines are increasingly providing answers rather than just search results, but that's a UI/pro…

>The entire comparison hinges on people only making simple factual searches You have a point but no it doesn't. The article already kind of addresses it, but Open AI had a pretty low loss in 2024 for the volume of usage they get. 5B seems like a lot until you realize chatgpt.com alone even in 2024 was one of the most visited sites on the planet each month with the vast majority of those visits being entirely free use…

I'll definitely buy that argument for OpenAI, but then why are Anthropic/XAI etc losing money? They don't have the same generous free tiers as OpenAI and yet they keep raising absurd amounts of money.

Re: LLMs are cheap

#172

Post by SoftTalker »

ceejayoz wrote: post #111
brookst wrote: post #74

Earlier quoted context omitted.

Just tried asking “what is the maximum carryon size for an American Airlines flight DFW-CDG” and it used a webs search, provided the correct answer, and provided links to both the airline and FAA sites. Why wouldn’t I use it like this?

That search query brings up https://www.aa.com/i18n/travel-info/baggage/carry-on-baggage... for the first result, which says "The total size of your carry-on, including the handles and wheels, cannot exceed 22 x 14 x 9 inches (56 x 36 x 23 cm) and must fit in the sizer at the airport." What benefit did the LLM add here, if you still had to vet the sources?

> What benefit did the LLM add here

Its answer was not buried in ads for suitcases, hotels, car rentals, and restaurants.

Re: LLMs are cheap

#173

Post by DarmokJalad1701 »

mupuff1234 wrote: post #154
raincole wrote: post #148

Earlier quoted context omitted.

> an API that is likely a loss-leader to grab market share (hosted LLM cloud models) Everyone just repeats this but I never buy it. There is literally a service that allows you to switch models and service providers seamlessly (openrouter). There is just no lock-in. It doesn't make any financial sense to "grab market share". If you sell something with UI, like ChatGPT (the web interface) or Cursor, sure. But selling…

Except they most likely do have a plan to make it harder to switch.

Who is "they"? It makes no sense for Openrouter to allow providers that do not conform to the API. They profit from the commission from the fees and not providing inference.

Re: LLMs are cheap

#174

Post by jenny91 »

dist-epoch wrote: post #36
exceptione wrote: post #26

Earlier quoted context omitted.

This is only the power usage?

Hardware spend also need to be amortized (over 1 year? 2 years?) Unless you cloud rent.

5 year amortization is pretty realistic I'd say. A100s (came out 2020Q1) are still in heavy use. (I think V100s from 2017Q3 are starting to be phased out a fair bit.)

Re: LLMs are cheap

#175

Post by dragontamer »

JimDabell wrote: post #101
xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

> you also don't have any evidence that they are profitable. Sure we do. Go to AWS or any other hosting provider and pay them for inference. You think AWS are going to subsidise your usage of somebody else’s models indefinitely? > All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achieve profitability yet. Yes, capex not opex. T…

Purchasing new GPUs is capex but depreciation of GPUs is opex.

There's still a cost, it's just thrown into the future.

Re: LLMs are cheap

#176

Post by Xmd5a »

Palmik wrote: post #124
xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

> API that is likely a loss-leader to grab market share (hosted LLM cloud models). I don't think so, not anymore. If you look at API providers that host open-source models, you will see that they have very healthy margin between their API cost and inference hardware cost (this is, of course, not the only cost) [1]. And that does not take into account any proprietary inference optimizations they have. As for closed-mo…

I use whisper to transcribe long conversations, and deploying the model myself on vastai is ten times cheaper than OpenAI's API offer.

Re: LLMs are cheap

#177

Post by skywhopper »

barrkel wrote: post #77
WhyIsItAlwaysHN wrote: post #65

There's something I don't get in this analysis. The queries for the LLM which were used to estimate costs don't make a lot of sense for LLMs. You would not ask an LLM to tell you the baggage size for a flight because there might be a rule added a week ago that changes this or the LLM might hallucinate the numbers. You would ask an LLM with web search included so it can find sources and ground the answer. This applies…

Oh contraire, I ask questions about recent things all the time, because the LLM will do a web search and read the web page - multiple pages - for me, and summarize it all. 4o will always do a web search for a pointedly current question, give references in the reply that can be checked, and if it didn't, you can tell it to search. o3 meanwhile will do many searches and look at the thing from multiple angles.

Yeah, the point is that this behavior uses a lot more tokens than the OP says is a “typical” LLM query.

Re: LLMs are cheap

#178

Post by otterley »

barrkel wrote: post #83
jeffbee wrote: post #9

Earlier quoted context omitted.

To make a billion dollars, I would simply sell a Coke to everyone in China. I have been giving away Coke in China and it is very popular, so I am sure this will work.

You joke, but for food and beverages, a stand in the supermarket giving the stuff away for free is a really common (and thus successful) tactic.

It’s successful for some, but not for everyone. People play roulette all the time but that doesn’t mean everyone other than the house is making a profit. (BTW supermarkets charge for promotional space.)

Re: LLMs are cheap

#179

Post by skywhopper »

SoftTalker wrote: post #172
ceejayoz wrote: post #111

Earlier quoted context omitted.

That search query brings up https://www.aa.com/i18n/travel-info/baggage/carry-on-baggage... for the first result, which says "The total size of your carry-on, including the handles and wheels, cannot exceed 22 x 14 x 9 inches (56 x 36 x 23 cm) and must fit in the sizer at the airport." What benefit did the LLM add here, if you still had to vet the sources?

> What benefit did the LLM add here Its answer was not buried in ads for suitcases, hotels, car rentals, and restaurants.

Really sad that we have made the web so obnoxious that people want to use complex AI tech to re-simplify it.

Re: LLMs are cheap

#180

Post by frotaur »

lhl wrote: post #146
ceejayoz wrote: post #109

Earlier quoted context omitted.

> You think AWS are going to subsidise your usage of somebody else’s models indefinitely? As with Costco's giant $5 roasted chickens, this is not solid evidence they're profitable. Loss-leaders exist.

Rather than speculating another option is to just measure things. I churned through billions of tokens for evals and synthetic data earlier this year, so I did some of that. On an H100 node, a Llama3 70B FP8 at concurrency=128 generated at about 0.4 J/token (this was estimating node power consumption and multiplying by a generous PUE, 1.2X or something like that) - it was still 120X cheaper than the 48 J/token estima…

My only question about this is the concurrency : is it really easy to leverage it when you need to serve to clients without much latency ? I don't know much about this.
Post reply on HN