Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

281–290 of 294 posts

Re: I want everything local – Building my offline AI workspace

#281

Post by jononor »

snoman wrote: post #262
doctoboggan wrote: post #59

Earlier quoted context omitted.

> A vector database for years of emails can easily exceed 50GB. In 2025 I would consider this a relatively meager requirement.

Take whatever you're indexing and make it 16-20x and that’s a good approximation of what the vector db’s total size is going to be.

Why is it like that, currently? There is no information added by a vector index compared to the original text. And the text is highly redundant and compressible with even lossless functions. Furthermore a vector index is already lossy and approximate. So conceptually it is at least possible to have an index that would be a fraction of the size of what is indexed?

Re: I want everything local – Building my offline AI workspace

#282

Post by glhaynes »

jychang wrote: post #200
wy1346 wrote: post #186

Earlier quoted context omitted.

This looks incredibly useful for making large-scale local AI truly practical.

This is annoyingly Apple-only though. Even though my main dev machine is a Macbook, this would be a LOT more useful if it was a Docker container. I'd still take a Docker container over an Apple container, because even though docker is not VM-level-secure, it's good enough for running local AI generated code. You don't need DEFCON Las Vegas levels of security for that. And also because Docker runs on my windows gaming…

I’m no expert on these things, but since Apple Containerization uses OCI images, I’d think you’d be able to sub in Docker (or Podman, etc) as the runtime pretty trivially. Like Podman, it uses a very similar command line interface to Docker’s.

Edit: Oh, I see now that Coderunner is Apple Containerization-specific.

Re: I want everything local – Building my offline AI workspace

#283

Post by apitman »

Aurornis wrote: post #250
shaky wrote: post #2

This is something that I think about quite a bit and am grateful for this write-up. The amount of friction to get privacy today is astounding.

> The amount of friction to get privacy today is astounding I don't understand this. It's easy to get a local LLM running with a couple commands in the terminal. There are multiple local LLM runners to choose from. This blog post introduces some additional tools for sandboxed code execution and browser automation, but you don't need those to get started with local LLMs. There are multiple local options. This one is e…

> It's easy to get a local LLM running

Easy for what percentage of people?

Re: I want everything local – Building my offline AI workspace

#284

Post by kalasoo »

Agree

I agree on this in every aspect

AI or any technology serve users locally will eventually empower users in a great manner because users can fully understand what they want.

Like a paper and pencil, which was not "cheap" in early history but eventually local. "AI" or any technology will function the same way eventually.

why?

1. free to run and create (free == cheap, free == uncensored) 2. ambient everywhere

Re: I want everything local – Building my offline AI workspace

#285

Post by kalasoo »

Agree

I agree on this in every aspect

AI or any technology serve users locally will eventually empower users in a great manner because users can fully understand what they want.

Like a paper and pencil, which was not "cheap" in early history but eventually "local". AI or any technology will function the same way eventually.

why?

1. free to run and create (free == cheap, free == uncensored) 2. ambient everywhere

Re: I want everything local – Building my offline AI workspace

#286

Post by mattlutze »

andylizf wrote: post #72
doctoboggan wrote: post #59

Earlier quoted context omitted.

> A vector database for years of emails can easily exceed 50GB. In 2025 I would consider this a relatively meager requirement.

Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that poi…

The DGX Spark being just $3-4,000 with 4TB of storage, 128GB unified memory, etc (or the Mac Studio tbh) is a great indicator that Local AI can soon be cheap and, along with the emerging routing and expert mixing strategies, incredibly performant for daily needs.

Re: I want everything local – Building my offline AI workspace

#287

Post by BoredPositron »

andylizf wrote: post #72
doctoboggan wrote: post #59

Earlier quoted context omitted.

> A vector database for years of emails can easily exceed 50GB. In 2025 I would consider this a relatively meager requirement.

Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that poi…

That's the size of just two or three triple A games nowadays.

Re: I want everything local – Building my offline AI workspace

#288

Post by 1oooqooq »

jazzypants wrote: post #117
1oooqooq wrote: post #76

Earlier quoted context omitted.

more interesting is the extent apple convinced people a laptop can replace a desktop or server. mind blowing reality distortion field (as will be proven by some twenty comments telling I'm wrong 3... 2... 1).

I think this would be more interesting if you were to try to prove yourself correct first. There are extremely few things that I cannot do on my laptop, and I have very little interest in those things. Why should I get a computer that doesn't have a screen? You do realize that, at this point of technological progress, the computer being attached to a keyboard and a screen is the only true distinguishing factor of a l…

cool. you can browse the web. that's cool. just stay out of conversation you're not an authority.

Re: I want everything local – Building my offline AI workspace

#289

Post by snoman »

jononor wrote: post #281
snoman wrote: post #262

Earlier quoted context omitted.

Take whatever you're indexing and make it 16-20x and that’s a good approximation of what the vector db’s total size is going to be.

Why is it like that, currently? There is no information added by a vector index compared to the original text. And the text is highly redundant and compressible with even lossless functions. Furthermore a vector index is already lossy and approximate. So conceptually it is at least possible to have an index that would be a fraction of the size of what is indexed?

There is some information added, depending on the vector db and context (some systems will add permissions related metadata so that the LLM won’t pull chunks that the user didn’t have access to).

The vector itself is pretty large (512 dimensions).

The chunks have an overlap (iirc 30% but someone feel free to correct me).

I don’t _think_ the data is typically compressed (not sure why but I assume performance).

Re: I want everything local – Building my offline AI workspace

#290

Post by rubymamis »

moffkalast wrote: post #205
senko wrote: post #198

Earlier quoted context omitted.

Cross-platform toolkits are (still) a thing.

Yeah they're called Electron now ;) Qt is such a pain to work with it's almost like it's intentional that people should avoid it.

I can't disagree more. I've written extensively about the joy of programming using Qt in my blog post: https://rubymamistvalove.com/block-editor
Post reply on HN