Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local AI needs to be the norm

unix.foo

Post reply on HN
Print view View original

581–590 of 804 posts

Re: Local AI needs to be the norm

#581

Post by StilesCrisis »

themafia wrote: post #494
crazygringo wrote: post #148

Earlier quoted context omitted.

I use an extension that gives me a customized homepage, but I still always get the "what's new" tab on every major version upgrade. It's a totally separate tab that opens. It's got nothing to do with what you use as your homepage.

Thank you for going out of your way to deny my exact experience. Do you think I'm doing this to rag on Google? And you're this eager to defend them? I'm on gentoo. I have to update chrome manually. I updated it. On update I _never_ get a "what's new" page. I've had this profile for more than a decade so I have no actual idea why, but, I can absolutely tell you, I do *not* get one. After update it started consuming al…

That makes sense. You aren't really "updating" at all. You're basically reinstalling a new Chrome on every update. It makes sense then that you aren't seeing "what's new" because that's not how a fresh install starts up.

Note that this package and update is actually not maintained by Google at all, it's done by Gentoo: https://wiki.gentoo.org/wiki/Project:Chromium/How_to_bump_Ch...

I hate to be an apologist for anything but I think you are pointing fingers in the wrong place. The Google-official releases use the built-in automatic updater and do show What's New. This is a Gentoo release and they chose to do their own thing for updates.

Re: Local AI needs to be the norm

#583

Post by Spooky23 »

ThunderSizzle wrote: post #283
Spooky23 wrote: post #245

Earlier quoted context omitted.

Is it so different? If the US’s fascist experiment continues past the current president, we’ll absolutely be nationalizing frontier companies or exerting equivalent control.

[flagged]

And Trump simps are real. I said continued fascism post-Trump.

Help me out, which major public company did the Obama administration take a material ownership stake in? Which media companies did George W pressure into putting toadies into key management and editorial positions?

I can't find those references, but perhaps the liberal media is hiding the truth?

Re: Local AI needs to be the norm

#584

Post by [deleted] »

stingraycharles wrote: post #562
adamtaylor_13 wrote: post #307

Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.

Opus is probably somewhere in the 5TB parameter range and needs terabytes of GPU memory. The economics of running SOTA locally just does not make sense, because you’re not using it 24/7 at 80%+ utilization while the cloud based providers can.

[deleted]

Re: Local AI needs to be the norm

#585

Post by noashavit »

Relying on external APIs network failure points and unavoidable latency from the round trips. There is also the AI API rate limits that come into play. We might find that for critical workflows, local compute is the only reliable architecture.

Re: Local AI needs to be the norm

#586

Post by appreciatorBus »

supermdguy wrote: post #306

Interesting to see this after the recent post about Chrome’s on-device model using up 4gb of storage, which frustrated a lot of people [1]. I agree local models are great, and it’s cool that Apple has models built in now. But I feel like it basically has to be an OS level feature or users are going to get upset. I’d certainly rather have a small utility call out to OpenAI than download its own model. [1]: https://new…

The way I interpret the drama over the Chrome model is that for a large chunk of users, perhaps the majority, Chrome is the OS, and this 4GB model will be their OS Level feature for local AI.

Re: Local AI needs to be the norm

#587

Post by bluGill »

WRONG, this completely ignores the most important issue and so is completely wrong.

The important issue is where is the data stored. And there are far to many advantages to having your data in the cloud: you can access it from whatever device you happen to have, and it isn't lost if you lose the device. This also outsources your backups to the cloud which is probably doing a much better job than you would (maybe no on hacker news, but nearly everyone else) - the cloud has earned a bad reputation for backups, but it is still much better than most people would be.

Once you accept the data is going to be elsewhere it doesn't matter if the compute is elsewhere or not. The data is the important part.

What needs to be the norm is more self-hosting your own data. Companies should not be outsourcing this by default - even where you outsource some of it, you need to watch your contracts and ensure the ownership is yours - not shared. Once your data is yours on your own cloud accessible servers we can start asking can we run our AI models in the same data center as we already have our data in. I don't need my AI model to run on my phone, it can run on the server in my basement which has a lot more power available (my phone has a better GPU but I can't afford the battery power to run AI on my phone)

Re: Local AI needs to be the norm

#588

Post by entrope »

zozbot234 wrote: post #556
entrope wrote: post #536

Earlier quoted context omitted.

What's the basis for saying local tokens will always be cheaper? As others have outlined, LLMs serving one user at a time are pretty expensive, but concurrent users become much more cost-effective (assuming there's enough RAM for the contexts). If "local" to you means ~10 hours daily use by a team of employees, the company still has to balance against cloud services that can amortize non-recurring costs over 24 hours…

Why would a team of employees not be able to run AI workloads 24/7? Not all workloads are time sensitive.

Both my experience, and Anthropic's off-peak promotion, indicate that there are very uneven levels of demand for peak hours versus off-peak hours. How close do you think they are?

Re: Local AI needs to be the norm

#589

Post by mohamedkoubaa »

bluGill wrote: post #587

WRONG, this completely ignores the most important issue and so is completely wrong. The important issue is where is the data stored. And there are far to many advantages to having your data in the cloud: you can access it from whatever device you happen to have, and it isn't lost if you lose the device. This also outsources your backups to the cloud which is probably doing a much better job than you would (maybe no o…

I want a way to backup my data fully encrypted somewhere and have custody of the keys - but importantly, the data should all be decrypted locally where all my apps can use the data without any network

Re: Local AI needs to be the norm

#590

Post by pcthrowaway »

bluGill wrote: post #587

WRONG, this completely ignores the most important issue and so is completely wrong. The important issue is where is the data stored. And there are far to many advantages to having your data in the cloud: you can access it from whatever device you happen to have, and it isn't lost if you lose the device. This also outsources your backups to the cloud which is probably doing a much better job than you would (maybe no o…

> What needs to be the norm is more self-hosting your own data

I assumed self-hosted AI would fall under local AI for the purposes of this article. Does the author really need to spell it out?

Post reply on HN