Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

21–30 of 294 posts

Re: I want everything local – Building my offline AI workspace

#21

Post by b112 »

jeremyjh wrote: post #16
tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

I expect it will never change. In two years if there is a local option as good as GPT-5 there will be a much better cloud option and you'll have the same tradeoffs to make.

I grew up in a time when listening to an mp3 was too computationally expensive and nigh impossible for the average desktop. Now tiny phones can decode high def video realtime due to CPU extensions.

And my phone uses a tiny, tiny amount of power, comparatively, to do so.

CPU extensions and other improvements will make AI a simple, tiny task. Many of the improvements will come from robotics.

Re: I want everything local – Building my offline AI workspace

#22

Post by frank_nitti »

sneak wrote: post #14
shaky wrote: post #2

This is something that I think about quite a bit and am grateful for this write-up. The amount of friction to get privacy today is astounding.

This writeup has nothing of the sort and is not helpful toward that goal.

I'd assume they are referring to being able to run your own workloads in a home-built system, rather then surrendering that ownership to the tech giants alone

Re: I want everything local – Building my offline AI workspace

#23

Post by pfannkuchen »

jeremyjh wrote: post #16
tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

I expect it will never change. In two years if there is a local option as good as GPT-5 there will be a much better cloud option and you'll have the same tradeoffs to make.

It might change once the companies switch away from lighting VC money on fire mode and switch to profit maximizing mode.

I remember Uber and AirBnB used to seem like unbelievably good deals, for example. That stopped eventually.

Re: I want everything local – Building my offline AI workspace

#24

Post by duxup »

jeremyjh wrote: post #16
tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

I expect it will never change. In two years if there is a local option as good as GPT-5 there will be a much better cloud option and you'll have the same tradeoffs to make.

Maybe, but my phone has become is a "good enough" computer for most tasks compared to a desktop or my laptop.

Seems plausible the same goes for AI.

Re: I want everything local – Building my offline AI workspace

#25

Post by hombre_fatal »

c-hendricks wrote: post #17
jeremyjh wrote: post #16

Earlier quoted context omitted.

I expect it will never change. In two years if there is a local option as good as GPT-5 there will be a much better cloud option and you'll have the same tradeoffs to make.

Why would AI be one of the few areas where locally-hosted options can't reach "good enough"?

For some use-cases, like making big complex changes to big complex important code or doing important research, you're pretty much always going to prefer the best model rather than leave intelligence on the table.

For other use-cases, like translations or basic queries, there's a "good enough".

Re: I want everything local – Building my offline AI workspace

#27

Post by kasey_junk »

jeremyjh wrote: post #16
tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

I expect it will never change. In two years if there is a local option as good as GPT-5 there will be a much better cloud option and you'll have the same tradeoffs to make.

I’d be surprised by that outcome. At one point databases were cutting edge tech with each engine leap frogging each other in capability. Still the proprietary db often have features that aren’t matched elsewhere.

But the open db got good enough that you need to justify not using them with specific reasons why.

That seems at least as likely an outcome for models as they continue to improve infinitely into the stars.

Re: I want everything local – Building my offline AI workspace

#28

Post by Imustaskforhelp »

braooo wrote: post #12
tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

Running LLMs at home is a repeat of the mess we make with "run a K8s cluster at home" thinking You're not OpenAI or Google. Just use pytorch, opencv, etc to build the small models you need. You don't need Docker even! You can share over a simple code based HTTP router app and pre-shared certs with friends. You're recreating the patterns required to manage a massive data center in 2-3 computers in your closet. That's…

So I love linux and would wish to learn devops one day in its entirety to be an expert to actually comment on the whole post but

I feel like they actually used docker for just the isolation part or as a sandbox (technically they didn't use docker but something similar to it for mac (apple containers) ) I don't think that it has anything to do with k8s or scalability or pre shared cert or http router :/

Re: I want everything local – Building my offline AI workspace

#29

Post by Imustaskforhelp »

frank_nitti wrote: post #22
sneak wrote: post #14

Earlier quoted context omitted.

This writeup has nothing of the sort and is not helpful toward that goal.

I'd assume they are referring to being able to run your own workloads in a home-built system, rather then surrendering that ownership to the tech giants alone

Also you get a sort of complete privacy that the data never leaves your home too whereas at best you would have to trust the AI cloud providers that they are not training or storing that data.

Its just more freedom and privacy in that matter.

Re: I want everything local – Building my offline AI workspace

#30

Post by xt00 »

Yea in an ideal world there would be a legal construct around AI agents in the cloud doing something on your behalf that could not be blocked by various stakeholders deciding they don't like the thing you are doing even if totally legal. Things that would be considered fair use, or maybe annoying to certain companies should not be easy for companies to just wholesale block by leveraging business relationships. Barring that, then yea, a local AI setup is the way to go.
Post reply on HN