Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

311–319 of 319 posts

Re: LLMs are cheap

#311

Post by indigodaddy »

Xmd5a wrote: post #176
Palmik wrote: post #124

Earlier quoted context omitted.

> API that is likely a loss-leader to grab market share (hosted LLM cloud models). I don't think so, not anymore. If you look at API providers that host open-source models, you will see that they have very healthy margin between their API cost and inference hardware cost (this is, of course, not the only cost) [1]. And that does not take into account any proprietary inference optimizations they have. As for closed-mo…

I use whisper to transcribe long conversations, and deploying the model myself on vastai is ten times cheaper than OpenAI's API offer.

I’m assuming doing transcription on a vast GPU is also ten+ times faster than local options?

https://news.ycombinator.com/item?id=44225953

Re: LLMs are cheap

#312

Post by noodletheworld »

vslira wrote: post #185
noodletheworld wrote: post #133

Earlier quoted context omitted.

I don’t completely disagree, but “assertion one” [1] [1] ~ you can obviously verify this yourself by doing it yourself and seeing how expensive it is. …is an enormously weak argument. You suppose. You guess. We guess. Let’s be honest, you can just stop at: > I don’t think so. Fair. I don’t either; but that’s about all we can really get at the moment afaik.

he's not wrong, if you can run a open weights model in any cloud, you can very straightforwardly estimate the cost of running the model. considering that these providers either use long-term contracts or maybe even buy their own hardware, this theoretical cloud deployment is itself an overestimate of the costs

…and its perfectly legit to run that, write the numbers down and link to it.

But:

A) it makes absolutely no difference to the fact you have no idea what the big LLM providers are actually doing.

B) Just asserting some random thing and saying “anyone competent can verify this themselves” is a weak argument. Youre saying youve done the research, but failing to provide any evidence you actual have

If youve crunched the numbers then man up and post them.

If not, then stop at “I think…”

“This is based on my experience running production workloads…” is a nice way of saying “I dont have any data to backup what Im saying”.

If you did, you could just link to it.

…by not posting data you make your argument non-falisifyable.

It is just an oppinion.

Re: LLMs are cheap

#313

Post by nitwit005 »

TZubiri wrote: post #296
nitwit005 wrote: post #272

Earlier quoted context omitted.

Brand is huge in every market. It's hard to get people to visit your website at all. People know about OpenAI, and look it up.

No network effect + already profitable. Not at all like Uber, let it go

Incoherent response.

Re: LLMs are cheap

#314

Post by Gerardo1 »

Spivak wrote: post #276
Gerardo1 wrote: post #238

Earlier quoted context omitted.

> You think AWS are going to subsidies your usage of somebody else’s models Yes >indefinitely? No, and that's the point.

Have they ever actually done this? I can't think of a time they've actually raised their prices ever that isn't the Route53 passing on registrar costs.

That assumes that they've been similarly situated with an offering that isn't profitable and has no path to profitability.

What company selling a primarily AI-based service right now is making a profit on that service?

Re: LLMs are cheap

#315

Post by lproven »

There is more to cost, and cheapness, than the direct financial cost.

The ecological cost is far more important.

Arguably, the impacts in employment, careers, theft of creative works, and other damage of inexpensive LLM bots are in the short term, and terms of impacts on mere human lives, more imminently and pressingly so.

Re: LLMs are cheap

#316

Post by dcre »

stavros wrote: post #310
qeternity wrote: post #308

Earlier quoted context omitted.

> LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities. I don't mean to be too pointed here, but it doesn't sound like you have built anything at scale with LLMs. They are absolutely not plug n play from a behavior perspective. Yes, there is API compatibility (text in, text out) but that is not what matter…

What kind of quirks have you seen that the next model wasn't better at?

A simple example would be when models get better at following instructions, the frantic and somewhat insane-sounding exhortations required to get the crappier model to do what you want can cause the stronger model to be a bit too literal and inflexible.

Re: LLMs are cheap

#317

Post by xigoi »

nonethewiser wrote: post #184
paxys wrote: post #20

The entire comparison hinges on people only making simple factual searches ("what is the capital of USA") on both search engines and LLMs. I'm going to say that's far enough from the standard use case for both these sets of APIs to be entirely meaningless. - If I'm using a search engine, I want to search the web. Yes these engines are increasingly providing answers rather than just search results, but that's a UI/pro…

>If I'm using a search engine, I want to search the web. Yes these engines are increasingly providing answers rather than just search results, but that's a UI/product feature rather than an API one. This is a great point, lets hold onto that. >If I'm using an LLM, it is for parsing large amounts of input data, image recognition, complex analysis, deep thinking/reasoning, coding. Strongly disagree. Sometimes when goog…

> I understand the idea that "if im googleing I want the index" but there is a reason google is increasingly burying their search results.

Yes. The reason being that Google does not want you to ure other websites than Google.

Re: LLMs are cheap

#318

Post by cush »

chipsrafferty wrote: post #289
cush wrote: post #287

Earlier quoted context omitted.

Sure. To clarify, I'm not asserting that in two years today's 4o will be 100x more expensive, but the sum of many core offerings from various companies will be. It won't be unheard of for people to spend $2k-$10k/yr between many AI services

It's not unheard of for people to spend $13,000 on a pure silver frying pan. Is it common? No.

Like say 10% of people or more

Re: LLMs are cheap

#319

Post by FarMcKon »

These comments are such an amazing show of very knowledgable and wise people, and complete moppets that have no clue, and can't stop playing expert on things they have no clue on.

:pop-corn:

Post reply on HN