Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

511–520 of 804 posts

Re: Local AI needs to be the norm

#511

Post by jillesvangurp »

I get the sentiment for self hosting. But there are a few counter arguments:

- Self hosting is expensive. It involves expensive machines with GPUs that cost hundreds per month if you use cloud based ones. You might need multiple of those. And you need people to mind those machines and they are even more expensive per month.

- If you run stuff on your laptop, it consumes a lot of resources and energy. I have qwen running on my laptop. Even minimal usage turns my laptop in a radiator. Nice as a demo, but I can't have it this hot all the time. It would run out of battery, and it's probably not great for longevity of components in the laptop.

- Models are evolving quickly and the self hosted smaller ones aren't as good when it comes to things like tool usage, reasoning, etc. Being able to switch tot he latest model is valuable.

- It's easier to get your use case working with one of the top models than with one of the smaller self hosted ones.

- If you get the wrong hardware, it might not be able to run the latest models very soon.

- Self hosting models is mostly a cost optimization. It only becomes relevant if you hit a certain scale.

- You have alternatives in the form of hosted models via a wide range of service providers. Some of those are EU based and offer all the things you'd be looking for if you are offering your services there. Including legal requirements.

- Reinventing what these companies do in house is technically challenging and possibly more expensive than self hosting models because now you need a lot of engineering capacity dedicated to that. And legal. And all the rest.

If, like most companies/people, you are at the experimenting stage, the cheapest and fastest is just getting an API key from an API provider of your choice. You can take it from there if your experiment actually works. And then it's mostly about optimizing cost. If your API usage goes to the thousands per month or worse, it becomes a cost/quality trade off.

Re: Local AI needs to be the norm

#512

Post by DrScientist »

duchenne wrote: post #357

Cloud models can use batch processing which is significantly more efficient. A local model has basically a batch of one which takes as much time to process as a batch of 100 because the gpu is memory bound and spend most of its time loading the model from vram to the gpu cache while the gpu cores are idle. With a batch of 100 the model loading time and compute time are roughly similar. So local Models have a first 10…

And what if your local computer essentially has an model chip with dedicated memory where the model stays loading 100% of the time?

Re: Local AI needs to be the norm

#513

Post by Galanwe »

root_axis wrote: post #346
doctorpangloss wrote: post #325

Earlier quoted context omitted.

Two Mac Studio M3 Ultra 512GB and 1 USB cable can run all those models - maybe about $30,000 in hardware - and based on my benchmarks, those Mac Studios were twice as fast as the A100s on Deepseek v4 Flash, which has a quantization but not really a lossy one.

That cannot run KimiK2.6 or GLM5.1 i.e models within the ballpark of anything offered by frontier companies.

Yes it can, but the experience is not great.

A single M3 maxed can run a Q2 Kimi 2.6, though thats with a hardly degraded perplexity.

2x M3s with RDMA can run a lossless Kimi2.6 at Q4, but with CPU only you would get okayish decode but horrible (+1m) TTFT, that wouldnt be a great _interactive_ experience.

Re: Local AI needs to be the norm

#514

Post by fennecfoxy »

satvikpendem wrote: post #333
0xbadcafebee wrote: post #284

Here's some things you can do right now with local models on a consumer device: - text-to-speech - speech-to-text - dictionary - encyclopedia - help troubleshooting errors - generate common recipes and nutritional facts - proofread emails, blog posts - search a large trove of documents, find information, summarize it (RAG) - manipulate your terminal/browser/etc - analyze a picture or video - generate a picture or vid…

Please add double new lines as your formatting for the bullet point list makes it all one paragraph.

Tbf I've always hated that about HN formatting as it's not very clear at all that that's how it works.

If there's a newline in my comment, why not retain it? Whyyyyy?!

Re: Local AI needs to be the norm

#515

Post by DrScientist »

DoctorOetker wrote: post #334

One advantage of local AI is continual learning. When I say 'moat' I don't mean moat specific to a company vis-a-vis other companies, but 'moat' specific to the set of inference providers vis-a-vis self-hosted local inference. The moat consists primarily of being able to batch inference requests. If we pretend people weren't interested in long context-lengths, there would be a moat for inference providers. who can ba…

Anybody know of good real world examples for continual learning?

Does it really work?

Re: Local AI needs to be the norm

#516

Post by fennecfoxy »

gkcnlr wrote: post #360

It seems like everybody is focused on "LLM"s, a.k.a Large Language Models. One interesting addition to that is fine-tuned- small parameter, distilled, context-dependent small language models that: 1- Do a particular task with great capability (due to its constrained, limited scope) 2- Do it in such a way, it integrates gracefully in your workflow without ever requiring you to know you are using an LM. There is a diff…

Eh I think the small model thing is kind of a no-go.

Reason being is that many workloads for AI are dynamically mixed, where training from multiple subjects comes into play and you just can't know exactly what mix will be required for each task ahead of time.

I was hoping loras would do this for us as well but they don't really seem to have worked out for llms (compared to in the image/video diffusion space).

Perhaps some future model will have some sort of "core" that can load/unload portions of itself dynamically at runtime. Like go for a very horizontal architecture/hundreds of MoE and unload/load those paths/weights once a parent value meets or exceeds some minimum, hmmm.

Re: Local AI needs to be the norm

#517

Post by xigoi »

fennecfoxy wrote: post #514
satvikpendem wrote: post #333

Earlier quoted context omitted.

Please add double new lines as your formatting for the bullet point list makes it all one paragraph.

Tbf I've always hated that about HN formatting as it's not very clear at all that that's how it works. If there's a newline in my comment, why not retain it? Whyyyyy?!

Because of the 6 people who write HN comments in Vim with hard wrapping turned on.

Re: Local AI needs to be the norm

#518

Post by teiferer »

Every reply here forgets/overlooks the main reason for why this is not going to happen: The astronomical AI data center investments currently underway. Those place are not just for training. They are for inference too and the way all those investments are expected to eventually pay off. The whole AI sector of our industry depends on running models in these places.

Re: Local AI needs to be the norm

#519

Post by maxdo »

The start of the argument is already broken . Ok , slapping api is bad , so you push api that mimics to your provider, install some Chinese llm that will never obey any lawsuit in your country , install 500 packages to do so , every of them has a potential risk a security issue . How is that better ?

Oh yeah , it feels independent and not lazy , sure

Re: Local AI needs to be the norm

#520

Post by Arn_Thor »

cpt_sobel wrote: post #467
pier25 wrote: post #258

Earlier quoted context omitted.

How fast do you reckon most people will be able to afford 128-256GB of RAM?

Their prices are currently so unreachable because of the big players hoarding every chip they can get their hands on, but if/when the market realizes that locally deployed LLMs are the way to go, maybe (hopefully?) then more chips will be available to the consumers for lower prices.

The only way that'll happen is if deep-pocketed corporate buyers exit the market almost entirely, and therefore stop being the highest-available bidder. Even in a scenario where it's obvious to everyone that consumer-side hardware is a viable option, it's still not in the big AI providers' interest to abandon the effort to push/pull everyone to their cloud. They'll keep buying as long as there's liquidity to fund them and the will to do so, and we're a ways off that collapsing. I'm quite pessimistic. Prices will probably come down in the next 12-18 months, but not to where they were before this
Post reply on HN