Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

591–600 of 804 posts

Re: Local AI needs to be the norm

#591

Post by andybak »

Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

> draw a pretty good pelican on a bike.

You mean the famously hard task? The one picked because it stretches frontier models to their limits?

Re: Local AI needs to be the norm

#592

Post by chasd00 »

mohamedkoubaa wrote: post #589
bluGill wrote: post #587

WRONG, this completely ignores the most important issue and so is completely wrong. The important issue is where is the data stored. And there are far to many advantages to having your data in the cloud: you can access it from whatever device you happen to have, and it isn't lost if you lose the device. This also outsources your backups to the cloud which is probably doing a much better job than you would (maybe no o…

I want a way to backup my data fully encrypted somewhere and have custody of the keys - but importantly, the data should all be decrypted locally where all my apps can use the data without any network

tar -czf - /path/to/folder | gpg -c -o folder.tar.gz.gpg then scp/POST that somewhere /s..kinda

Re: Local AI needs to be the norm

#594

Post by [deleted] »

Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

[deleted]

Re: Local AI needs to be the norm

#595

Post by numpad0 »

DennisP wrote: post #304
pier25 wrote: post #270

Earlier quoted context omitted.

I'm sure it will happen but I don't think it will be soon. 10 years ago I was using 16GB in my MBP and today it's 48GB. It's just a 3x increase during mostly a bonanza period.

For most of that time, I don't think many people had much use for more ram than that. If demand picks up, companies will provide it. And the Mac Studio was available with 512GB until ram got scarce and they cut the max in half recently.

IMO that was a really weird choice that everyone seemed to make. DDR5 2x64GB before the spike was like $250. I had not much justification to NOT go with 64GB for my pre-COVID build.

It seems that a lot of PC building people are confused too deeply by Intel marketing and fixated on getting the flashiest CPU attainable within budget. Similar things happened with previous AI hype, and some people were using HDD boot drives on GPU rigs and asking others whether low end i7 would cut it. They acted very confused when told that they need SSD and Pentium is plentium.

I mean, there is a shortage going on, but when it'll be over anyhow - whether due to all the last three standing filing bankruptcy or CXMT-Huawei starts delivering in shiploads or Kioxia enters the market - and it comes back down to $2/GB, or even $5/GB, just max it out and forget about it for 10 years. Why not.

Re: Local AI needs to be the norm

#596

Post by AdeelKhamisa »

Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

I was writing just about this last week for fun: AI + hardware team-up to build localized AI with specialized functions to your organization. Ex. Adoble Studio AI in an on premise Box, made by Apple and powered by something like Cohere with privacy:

https://www.notion.so/adeelkhamisa/Cohere-s-next-steps-to-be...

Re: Local AI needs to be the norm

#597

Post by tristor »

The biggest challenge I have with local models right now (and I use them extensively) is search integration and tool calling. The thing that Claude and ChatGPT get right for most general purpose use cases which is hard to do with a local model is the model deciding when to search vs use its built-in training, and having strong search tooling, as well as tool calling for additional data sources via MCP. If you can incorporate the right data into the context window, local models are more than good enough for general purpose usage as they stand today. Qwen 3.5, Gemma 4, even gpt-oss-120b are solid at reasonable quants if they have the right data.

The moment we see standardized and batteries-included pathways to integrate search, ideally at no additional cost, in things like LM Studio combined with better tool calling in the local models, you'll quickly see local model performance catch up.

Re: Local AI needs to be the norm

#598

Post by senko »

I love this line:

> Stop shipping distributed systems when you meant to ship a feature.

But not in the contex the author meant.

Many people don't realize that when you have a frontend, a backend (several instances, for failover/scaling), a (separate) database, maybe some object store -- you have a distributed system.

A recent article[0] touched on that, although most HN commenters[1] latched on the "go" part. But there's something to avoiding rube goldberg machines where we don't need them.

[0] https://blainsmith.com/articles/just-fucking-use-go/

[1] https://news.ycombinator.com/item?id=48062997

Re: Local AI needs to be the norm

#600

Post by henry_kang »

Akuehne wrote: post #578

I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. Wi…

Actually you can do way more things than that. We have optimized it to process 2TB of high def videos on a M5 MBP in under 24 hours, including everything such as speech understanding, face recog, LLM and VLM. Super fun.
Post reply on HN