Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

301–310 of 804 posts

Re: Local AI needs to be the norm

#301

Post by CuriouslyC »

ai_fry_ur_brain wrote: post #209
NitpickLawyer wrote: post #159

Earlier quoted context omitted.

API prices are most likely not subsidised. A brief look at openrouter can tell you that. There are plenty of providers that have 0 reason to subsidise that sell models at roughly the same average price. So the model works for them (or they wouldn't do it otherwise).

They are subsidized, heavily. This is simple math, there are lots of reasons to subsidize. Please go look up the hardware requirements to run your favorite model and a given tok/ps then multiple that by 86400 (seconds in a day) then divide that by 1mm and multiple by the $ per mm tokens, then ask yourself if there's any possibility they could be profitable or even close to break even. You are going off vibes alone, t…

Anthropic and OpenAI make money on API calls, margins have been reported in public filings. Subs are subsidized.

Re: Local AI needs to be the norm

#302

Post by D2OQZG8l5BI1S06 »

try-working wrote: post #200
2ndorderthought wrote: post #106

Earlier quoted context omitted.

I disagree. I think deepseek, qwen, and kimi earn a lot of trust open sourcing their models. While still profiting. Effectively they are saying "yea don't crowd our data centers with small queries, go ahead and send your frontier questions to our frontier models. Oh btw those us models? You can run something about as good for free from us if you want hah." It's a power and marketing move. It's also insanely smart to…

Correct. Open source is a PR and marketing strategy for new labs, regardless of origin. https://try.works/#why-chinese-ai-labs-went-open-and-will-re...

Interesting article, but Qwen does seem to be closing off. They don't release big variants anymore, and I'm not sure that the fact the local-LLM community keeps praising it actually increases the number of people using their API.

It did work for Deepseek for sure and it seems to move the needle for Xiaomi's MiMo; but will it be enough for Qwen and Gemma? Those are the models you can actually run without going all-in on AI (but only with gaming GPUs and such).

Re: Local AI needs to be the norm

#303

Post by try-working »

I'm building a protocol and router runtime for hybrid local/cloud AI.

The goal is that you would assign roles to models based on tasks, capabilities and observed performance. The router would then take care of model selection in the background.

It's tricky though. Probably have another two weeks before I can release the runtime.

I have a preview up at https://role-model.dev/

You can follow me on Twitter if you want updates (see profile)

Re: Local AI needs to be the norm

#304

Post by DennisP »

pier25 wrote: post #270
Schiendelman wrote: post #267

Earlier quoted context omitted.

Other than this recent spike, it's been trending cheaper continuously for decades. In a few years 128GB will be as affordable as 12GB (what flagship phones have now) is today.

I'm sure it will happen but I don't think it will be soon. 10 years ago I was using 16GB in my MBP and today it's 48GB. It's just a 3x increase during mostly a bonanza period.

For most of that time, I don't think many people had much use for more ram than that. If demand picks up, companies will provide it.

And the Mac Studio was available with 512GB until ram got scarce and they cut the max in half recently.

Re: Local AI needs to be the norm

#305

Post by try-working »

D2OQZG8l5BI1S06 wrote: post #302
try-working wrote: post #200

Earlier quoted context omitted.

Correct. Open source is a PR and marketing strategy for new labs, regardless of origin. https://try.works/#why-chinese-ai-labs-went-open-and-will-re...

Interesting article, but Qwen does seem to be closing off. They don't release big variants anymore, and I'm not sure that the fact the local-LLM community keeps praising it actually increases the number of people using their API. It did work for Deepseek for sure and it seems to move the needle for Xiaomi's MiMo; but will it be enough for Qwen and Gemma? Those are the models you can actually run without going all-in…

Definitely. Open releases will accelerate this year, including from Qwen because they're behind in adoption.

Re: Local AI needs to be the norm

#306

Post by supermdguy »

Interesting to see this after the recent post about Chrome’s on-device model using up 4gb of storage, which frustrated a lot of people [1].

I agree local models are great, and it’s cool that Apple has models built in now. But I feel like it basically has to be an OS level feature or users are going to get upset. I’d certainly rather have a small utility call out to OpenAI than download its own model.

[1]: https://news.ycombinator.com/item?id=48019219

Re: Local AI needs to be the norm

#307

Post by adamtaylor_13 »

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you.

Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

Re: Local AI needs to be the norm

#308

Post by root_axis »

wincy wrote: post #294
root_axis wrote: post #290

Earlier quoted context omitted.

You are greatly underestimating the hardware requirements for productive local LLMs. Research consistently shows that parameter count sets the practical ceiling for a model's reliability. Quantized models with double digit param counts will never be reliable enough to achieve results in the realm of something like Opus 4.6.

Won’t these H100s drop in price in a few years? With the data center build out surely these will become 1/10th the price and you’ll be able to set up a local LLM as good as opus 4.7. Even if the frontier model become more advanced, and memory hungry, you could use the same power usage as your oven to run a current day frontier model as needed? If I could drop $10,000 to have an effectively permanent opus 4.7 subscrip…

> Won’t these H100s drop in price in a few years

Doubtful. The increase in demand is greatly outpacing supply, and all signs point to a continued acceleration in demand

> If I could drop $10,000 to have an effectively permanent opus 4.7 subscription today, I would.

lol well obviously, but realistically that price point is going to be closer to $100k, with a perpetual $1k a month in power costs.

Re: Local AI needs to be the norm

#309

Post by binyu »

adamtaylor_13 wrote: post #307

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

DeepSeek V4 with 1 million token context window is pretty powerful, although still not there. There's hope that Opus 4.5 level performance locally is not that far away.

Re: Local AI needs to be the norm

#310

Post by thefounder »

adamtaylor_13 wrote: post #307

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

Next year there will be Opus 4.5 level available on open source models so theoretically you may be able to run it locally but in reality it will be too expensive (i.e maybe 2 x max Studio 512GB ram each) for “normal” users.
Post reply on HN