Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

141–150 of 294 posts

Re: I want everything local – Building my offline AI workspace

#141

Post by mccoyb »

andylizf wrote: post #53

This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails c…

That can't be the correct paper...

I think you meant this: https://arxiv.org/abs/2506.08276

Re: I want everything local – Building my offline AI workspace

#142

Post by johnfn »

mccoyb wrote: post #141
andylizf wrote: post #53

This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails c…

That can't be the correct paper... I think you meant this: https://arxiv.org/abs/2506.08276

No no, getting your entire workflow local requires solving P=NP.

Re: I want everything local – Building my offline AI workspace

#143

Post by vFunct »

jauntywundrkind wrote: post #78
Aurornis wrote: post #47

Earlier quoted context omitted.

I think the local LLM scene is very fun and I enjoy following what people do. However every time I run local models on my MacBook Pro with a ton of RAM, I’m reminded of the gap between local hosted models and the frontier models that I can get for $20/month or nominal price per token from different providers. The difference in speed and quality is massive. The current local models are very impressive, but they’re sti…

I agree and disagree. Many of the best models are open source, just too big to run for most people. And there are plenty of ways to fit these models! A Mac Studio M3 Ultra with 512 GB unified memory though has huge capacity, and a decent chunk of bandwidth (800GB/s. Compare vs a 5090's ~1800GB/s). $10k is a lot of money, but that ability to fit these very large models & get quality results is very impressive. Perform…

The game changer technology that'll enable full 1TB+ LLM models for cheap is Sandisk's High Bandwidth Flash. Expect devices with that in about 3-4 years, maybe even on cellphones.

Re: I want everything local – Building my offline AI workspace

#144

Post by jki275 »

davidmurdoch wrote: post #138
1oooqooq wrote: post #76

Earlier quoted context omitted.

more interesting is the extent apple convinced people a laptop can replace a desktop or server. mind blowing reality distortion field (as will be proven by some twenty comments telling I'm wrong 3... 2... 1).

I dropped $4k on an (Intel) laptop a few years ago. I thought it would blow my old 2012 core i7 out of the water. Editing photos in Lightroom and Photoshop often requires heavy sustained CPU work. Thermals in laptops is just not a solved problem. People who say laptops are fine replacements for desktops probably don't realize how much and how quickly thermals limit heavy multi-core CPU workloads.

That was true until Apple released the M series laptops.

Re: I want everything local – Building my offline AI workspace

#145

Post by unixhero »

andylizf wrote: post #53

This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails c…

I have 26tb hardrives, 50gb doesnt scare me. Or should I be?

Re: I want everything local – Building my offline AI workspace

#146

Post by bee_rider »

tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

Hardware is slower to design and manufacture than we expect as software people.

What I think we’ll see is: people will realize some things that suck in the current first-generation of laptop NPUs. The next generation of that hardware will get better as a result. The software should generally get better and lighter. We’re currently at step -.5 here, because ~nobody has bought these laptops yet! This will happen in a couple years.

Meanwhile, eventually the cloud LLM hosts will run out of investors money to subsidize our use of their computers. They’ll have to actually start charging enough to make a profit. On top of what local LLM folks have to pay, the cloud folks will have to pay:

* Their investors

* Their security folks

* The disposal costs for all those obsolete NVIDIA cards

Plus the remote LLM companies will have the fundamental disadvantage that your helpful buddy that you use as a psychologist in a pinch is also reporting all your darkest fears to Microsoft or whoever. Or your dev tools might be recycling all the work you thought you were doing for your job, back into their training set. And might be turned off. It just seems wildly unappealing.

Re: I want everything local – Building my offline AI workspace

#147

Post by esseph »

jauntywundrkind wrote: post #128
esseph wrote: post #95

Earlier quoted context omitted.

https://pcisig.com/pci-sig-announces-pcie-80-specification-t... From 2003-2016, 13 years, we had PCIE 1,2,3. 2017 - PCIE 4.0 2019 - PCIE 5.0 2022 - PCIE 6.0 2025 - PCIE 7.0 2028 - PCIE 8.0 Manufacturing and vendors are having a hard time keeping up. And the PCIE 5.0 memory is.. not always the most stable.

Thanks for the numbers. Valuable contribution for sure!! There's been a huge lag for PCIe adoption, and imo so so much has boiled down "do people need it"? In the past 10 years I feel like my eyes have been opened that every high tech company's greatest highest most compelling desire is to slow walk the release out. To move as slow as the market will bear, to do as little as possible, to roll on and on with minor inc…

All of these extremely high end technologies are so far away from hitting the consumer market.

Is there any desire for most people? What's the TAM?

Re: I want everything local – Building my offline AI workspace

#149

Post by notsylver »

andylizf wrote: post #72
doctoboggan wrote: post #59

Earlier quoted context omitted.

> A vector database for years of emails can easily exceed 50GB. In 2025 I would consider this a relatively meager requirement.

Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that poi…

You already need very high end hardware to run useful local LLMs, I don't know if a 200gb vector database will be the dealbreaker in that scenario. But I wonder how small you could get it with compression and quantization on top
Post reply on HN