Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

191–200 of 294 posts

Re: I want everything local – Building my offline AI workspace

#191

Post by justincormack »

jauntywundrkind wrote: post #163
esseph wrote: post #147

Earlier quoted context omitted.

All of these extremely high end technologies are so far away from hitting the consumer market. Is there any desire for most people? What's the TAM?

Classic economics thinking: totally fucked "faster horses" thinking. The addressable market depends on the advantage. Which right now: we don't know. It's all a guess that someone is going to find it valuable, and no one knows. But if we find that we didn't actually need $700 NIC's to get shitty bandwidth, if we could have just been putting cables from PCIe shaped slot to PCIe slot (or oculink port!) and getting >>10…

Thunderbolt tunnels pcie and you can use it as a nic in effect with one cable between devices. Its slower than oculink but more convenient.

Re: I want everything local – Building my offline AI workspace

#192

Post by typpilol »

spauldo wrote: post #169
thorncorona wrote: post #92

Earlier quoted context omitted.

I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?

A lot of us just don't want to be web developers. I mostly write IEC 61131 code, with sprinkles of BASIC (yuck), C, Perl, and Lisp. I've used JavaScript and quite frankly, you can keep it.

Does anyone else think javascript bad? Wow brave!

Re: I want everything local – Building my offline AI workspace

#193

Post by esseph »

r_lee wrote: post #190
esseph wrote: post #182

Earlier quoted context omitted.

Not sure I agree with that. It's a lot better on battery life and superior experience, especially if you are traveling or around areas with bad cell service. Cookies track me around on websites all the time + modern telemetry is pretty crazy.

The permissions that apps can get on Android even by default are pretty invasive, like querying other apps/processes and etc iirc...

Chrome being able to scan your network on desktop is still insane to me.

Re: I want everything local – Building my offline AI workspace

#194

Post by brabel »

PeterStuer wrote: post #173
andylizf wrote: post #72

Earlier quoted context omitted.

Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that poi…

While your aims are undoutably sincere, in practice for the 'local ai' target people building their own rigs usually have. 4TB or more fast ssd storage. The bottom tier (not meant disparagingly) are people running diffusion models as these do not have the high vram requirements. They generate tons of images or video, going form a one-click instally like Easydiffusion to very sophisticated workflows in comfyui. For th…

Speak for yourself! If it took me 500GB to store my vectors , on top of all my existing data, it would be a huge barrier for me.

Re: I want everything local – Building my offline AI workspace

#195

Post by hdgvhicv »

brabel wrote: post #194
PeterStuer wrote: post #173

Earlier quoted context omitted.

While your aims are undoutably sincere, in practice for the 'local ai' target people building their own rigs usually have. 4TB or more fast ssd storage. The bottom tier (not meant disparagingly) are people running diffusion models as these do not have the high vram requirements. They generate tons of images or video, going form a one-click instally like Easydiffusion to very sophisticated workflows in comfyui. For th…

Speak for yourself! If it took me 500GB to store my vectors , on top of all my existing data, it would be a huge barrier for me.

A 4tb external drive is £100. A 1TB sd card or usb stick a similar cost.

Maybe Im too old to appreciate what “fast” means, but storage doesnt seem an enormous cost once you stripe it.

Re: I want everything local – Building my offline AI workspace

#196

Post by justincormack »

mxmlnkn wrote: post #109
Uehreka wrote: post #70

Earlier quoted context omitted.

I was talking about this in another comment, and I think the big issue at the moment is that a lot of the local models seem to really struggle with tool calling. Like, just straight up can’t do it even though they’re advertised as being able to. Most of the models I’ve tried with Goose (models which say they can do tool calls) will respond to my questions about a codebase with “I don’t have any ability to read files,…

This resonates. I have finally started looking into local inference a bit more recently. I have tried Cursor a bit, and whatever it used worked somewhat alright to generate a starting point for a feature and for a large refactor and break through writer's blocks. It was fun to see it behave similarly to my workflow by creating step-by-step plans before doing work, then searching for functions to look for locations an…

You might end up using batching to run multiple queries or branches for yourself in parallel. But yes as you say it is very unclear right now.

Re: I want everything local – Building my offline AI workspace

#197

Post by moffkalast »

esseph wrote: post #152
thorncorona wrote: post #92

Earlier quoted context omitted.

I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?

Because the browser is gross and you can reclaim lot of performance and security when you don't need to use it.

Sure but you're also constrained to only one platform. It's like the C++ vs Python argument in ML, yes writing everything in low level high speed highly optimized native code would be perfect, but ain't (almost) nobody got fucking time or skill for that.

Re: I want everything local – Building my offline AI workspace

#198

Post by senko »

moffkalast wrote: post #197
esseph wrote: post #152

Earlier quoted context omitted.

Because the browser is gross and you can reclaim lot of performance and security when you don't need to use it.

Sure but you're also constrained to only one platform. It's like the C++ vs Python argument in ML, yes writing everything in low level high speed highly optimized native code would be perfect, but ain't (almost) nobody got fucking time or skill for that.

Cross-platform toolkits are (still) a thing.

Re: I want everything local – Building my offline AI workspace

#199

Post by anthk »

thorncorona wrote: post #92
com2kid wrote: post #55

> Even with help from the "world's best" LLMs, things didn't go quite as smoothly as we had expected. They hallucinated steps, missed platform-specific quirks, and often left us worse off. This shows how little native app training data is even available. People rarely write blog posts about designing native apps, long winded medium tutorials don't exist, heck even the number of open source projects for native desktop…

I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?

Offices when the performance matters against shitty web apps.

Re: I want everything local – Building my offline AI workspace

#200

Post by jychang »

wy1346 wrote: post #186
andylizf wrote: post #53

This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails c…

This looks incredibly useful for making large-scale local AI truly practical.

This is annoyingly Apple-only though. Even though my main dev machine is a Macbook, this would be a LOT more useful if it was a Docker container.

I'd still take a Docker container over an Apple container, because even though docker is not VM-level-secure, it's good enough for running local AI generated code. You don't need DEFCON Las Vegas levels of security for that.

And also because Docker runs on my windows gaming machine with a fast GPU with WSL ubuntu, and my linux VPS in the cloud running my website, etc etc. And most people have already memorized all the basic Docker commands.

This would be a LOT better if it was just a single docker command we can copy paste, run it a few times, and then delete if necessary.

Post reply on HN