Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

201–210 of 319 posts

Re: LLMs are cheap

#201

Post by lhl »

frotaur wrote: post #180
lhl wrote: post #146

Earlier quoted context omitted.

Rather than speculating another option is to just measure things. I churned through billions of tokens for evals and synthetic data earlier this year, so I did some of that. On an H100 node, a Llama3 70B FP8 at concurrency=128 generated at about 0.4 J/token (this was estimating node power consumption and multiplying by a generous PUE, 1.2X or something like that) - it was still 120X cheaper than the 48 J/token estima…

My only question about this is the concurrency : is it really easy to leverage it when you need to serve to clients without much latency ? I don't know much about this.

Yeah, actually for my batch usage, I usually push to 256+ concurrency, but on H100s at least, currently 64-128 is about the bend of the curve for where latency starts going out of control (this depends a lot on your context length and kvcache optimizations, though).

What I do for testing is that I will run a benchmark_serving sweep (I prefer ShareGPT for a standard set that is slightly more realistic for caching) with desired concurrency (eg 4-1024 or something like that) and then plot TTFT vs Total Throughput and graph Mean, P50, and P99 - this will give you a clear picture what your concurrency/throughput for a given desired latency.

Re: LLMs are cheap

#202

Post by seventhtiger »

ceejayoz wrote: post #200
seventhtiger wrote: post #197

Earlier quoted context omitted.

I had to accept cookies once, not each time I look up a recipe or a new piece of information. That's comparable to having to install a browser. I also didn't have to scan a hostile list of websites fighting for my attention to pick the correct one. It does that for me. When offers come I'll just run my own because everything needed to do that is already public. I'll never go back to the hell built by SEO and dark UX…

> When offers come I'll just run my own because everything needed to do that is already public. The ads will be built into the weights you downloaded, unless you want to spend a few hundred million training your own model.

The weights that are public today are already good enough for this. The cat is fully out of the bag.

Re: LLMs are cheap

#203

Post by handfuloflight »

loudmax wrote: post #198
handfuloflight wrote: post #187

Earlier quoted context omitted.

Why do you equate contextual with insidious?

The OP is not equating contextual with insidious. They're pointing out, correctly, that contextual ads can be insidious. And if they're profitable, they probably will be. A lot of the companies offering LLM services are in a race gain market share and build expertise. Right now, they can burn through millions of dollars of VC money, with the expectation that they'll turn a profit at some point in the future. If that…

> If that profit comes from advertising, and critically, if users don't expect advertising in their free LLMs, because they didn't see ads in generated output in the past, that will be very insidious.

Are the free LLM providers offering their service with a contractual obligation to the users that they will not add advertising to the outputs? If not, how is it insidious?

What definition of insidious are you using per https://www.merriam-webster.com/dictionary/insidious?

Re: LLMs are cheap

#204

Post by handfuloflight »

Workaccount2 wrote: post #199
handfuloflight wrote: post #187

Earlier quoted context omitted.

Why do you equate contextual with insidious?

Because then the AI isn't working for you anymore, it's working for the advertisers. Which isn't necessarily bad, but we can be pretty confident that the AI will not be upfront about this, and instead try to act like it's working for you.

If the advertising is contextually relevant, how is it working against you?

Re: LLMs are cheap

#205

Post by mediaman »

rco8786 wrote: post #170
JimDabell wrote: post #125

Earlier quoted context omitted.

OpenAI spends less on training than inference, so the worst case scenario is less than double the cost after factoring in training. Inference is still cheap.

Inference is cheap. Training is cheaper. Then where's all the money going? OpenAI is reporting heavy losses, but you're saying the unit economics of inference are all good. What are they spending money on?

Salary, mostly. It's useful to separate out the GPU cost of training from the salary cost of the people who design the training systems. They are expensive.

That does not mean, however, that inference is unprofitable. The unit economics of inference can be profitable even while the personnel costs of training next-generation models are extraordinary.

Re: LLMs are cheap

#206

Post by energy123 »

prmoustache wrote: post #15

LLMs aren't cheap if you consider the impact on the climate and the cost that comes from it.

Degrowth is a losing political argument.

You aren't getting what you want and you're helping the arsonists win elections by going with this strategy.

The winning argument is sustainable high growth with renewable energy.

Re: LLMs are cheap

#207

Post by ceejayoz »

seventhtiger wrote: post #202
ceejayoz wrote: post #200

Earlier quoted context omitted.

> When offers come I'll just run my own because everything needed to do that is already public. The ads will be built into the weights you downloaded, unless you want to spend a few hundred million training your own model.

The weights that are public today are already good enough for this. The cat is fully out of the bag.

I am heartened to discover we have finished the search for knowledge and no longer need any new info.

Re: LLMs are cheap

#208

Post by jsnell »

rco8786 wrote: post #170
JimDabell wrote: post #125

Earlier quoted context omitted.

OpenAI spends less on training than inference, so the worst case scenario is less than double the cost after factoring in training. Inference is still cheap.

Inference is cheap. Training is cheaper. Then where's all the money going? OpenAI is reporting heavy losses, but you're saying the unit economics of inference are all good. What are they spending money on?

Their spending is not a problem. It's quite low for a top-tier hard tech company that's also running a consumer service with 500M active users. They are making a loss because 95% of their users are on free accounts, and for now they're choosing not to monetize those users in any way (e.g. ads).

Re: LLMs are cheap

#209

Post by seventhtiger »

ceejayoz wrote: post #207
seventhtiger wrote: post #202

Earlier quoted context omitted.

The weights that are public today are already good enough for this. The cat is fully out of the bag.

I am heartened to discover we have finished the search for knowledge and no longer need any new info.

We just got a tool to circumvent advertisement and malicious diversion and influence.

Re: LLMs are cheap

#210

Post by slt2021 »

Workaccount2 wrote: post #106
xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

Just wait till there are ads for free users, which is going to happen. Depending on how insidious these ads are, they could be extremely profitable too, like recommending products and services directly in context.

most likely you will be targeted with ads based on what you give to the model. if you ask chatgpt about electric cars, expect a wave of ads coming at you from EV automakers from all channels: socials, media, email, mail, etc - trying to close you on their car brand
Post reply on HN