Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

231–240 of 294 posts

Re: I want everything local – Building my offline AI workspace

#231

Post by zozbot234 »

t_mann wrote: post #224

Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great f…

What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project.

Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.

Re: I want everything local – Building my offline AI workspace

#232

Post by [deleted] »

ricardobeat wrote: post #209
PeterStuer wrote: post #180

Earlier quoted context omitted.

Of course. And I am not arguing against that at all. Just like if someone makes an inference runtime that is 4% faster, I'll take that win. But would it be the decisive factor in my choice? Only if that was my bottleneck, my true constraint. All I tried to convey was that for most of the people in the presented scenario (personal emails etc.) , a 50 or even 500GB storage requirement is not going to be that primary co…

You are glossing over the fact that for RAG you need to search over those 500GB+ which will be painfully slow and CPU-intensive. The goal is fast retrieval to add data to the LLM context. Storage space is not the sole reason to minimize the DB size.

[deleted]

Re: I want everything local – Building my offline AI workspace

#233

Post by graemep »

zozbot234 wrote: post #231
t_mann wrote: post #224

Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great f…

What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project. Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.

You still need a lot of RAM though right? so its not going to be that cheap?

What sort of specs do you need?

Re: I want everything local – Building my offline AI workspace

#234

Post by xandrius »

brabel wrote: post #194
PeterStuer wrote: post #173

Earlier quoted context omitted.

While your aims are undoutably sincere, in practice for the 'local ai' target people building their own rigs usually have. 4TB or more fast ssd storage. The bottom tier (not meant disparagingly) are people running diffusion models as these do not have the high vram requirements. They generate tons of images or video, going form a one-click instally like Easydiffusion to very sophisticated workflows in comfyui. For th…

Speak for yourself! If it took me 500GB to store my vectors , on top of all my existing data, it would be a huge barrier for me.

Maybe time to update your storage?

Re: I want everything local – Building my offline AI workspace

#235

Post by cheschire »

oblio wrote: post #218
cheschire wrote: post #206

Earlier quoted context omitted.

But you would pump your secrets into a docker AI?

Does Docker do that or are you speculating? Also - podman?

I wasn't implying docker itself was the issue.

The previous commenter said that they didn't want to run a shell script that does "god knows what". The implication being that they would not trust the writer of the shell script.

They wanted a docker container that would setup this offline AI workspace for them, presumably so they could interact with the AI and feed "secrets" or otherwise private data into it. Obviously there are other use cases for an offline AI, but folks tend to let their guard down when they think something is offline-only, and they may not be as careful with .env values, or personal information, as they would with a SaaS frontier model.

So I was pointing out that the contents of the docker container would be also doing "god knows what" with their data. Sure they would get the offline user experience but then what happens? More shell scripts? Background data calls? etc. And of course it depends on how they configure their docker container, but if they aren't willing to review an install shell script, they probably aren't looking to do any level of effort for configuring Docker.

Hopefully that clarifies it.

Re: I want everything local – Building my offline AI workspace

#236

Post by wafflemaker »

notsylver wrote: post #149
andylizf wrote: post #72

Earlier quoted context omitted.

Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that poi…

You already need very high end hardware to run useful local LLMs, I don't know if a 200gb vector database will be the dealbreaker in that scenario. But I wonder how small you could get it with compression and quantization on top

I'm no dev either and still set up remote ssh login to be able to use LaTeX at home PC from my laptop.

Also, with many games and dual boot on my gaming PC I still have some space left on my 2TB NVME SSD. And my not enthusiast MOBO could fit two more.

It took so much time to install LaTeX and packages, and also so much space, my 128GB drive couldn't handle it.

Re: I want everything local – Building my offline AI workspace

#237

Post by mathiaspoint »

brbcompiling wrote: post #229

Local AI is awesome, but without beefy hardware it’s like trying to run a marathon in flip-flops.

If you have good flipflops you can walk miles without having to take a break. The other year Walmart had some really good George brand flipflops I used to wear everywhere.

Re: I want everything local – Building my offline AI workspace

#238

Post by jdelsman »

bamboozled wrote: post #228
OldfieldFund wrote: post #215

Earlier quoted context omitted.

I'm gonna put it here for visibility: Use patchright instead of Playwright: https://github.com/Kaliiiiiiiiii-Vinyzu/patchright

What problem does patchright solve?

Not being detected by things like bot detection.

Re: I want everything local – Building my offline AI workspace

#239

Post by mwcz »

notsylver wrote: post #149
andylizf wrote: post #72

Earlier quoted context omitted.

Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that poi…

You already need very high end hardware to run useful local LLMs, I don't know if a 200gb vector database will be the dealbreaker in that scenario. But I wonder how small you could get it with compression and quantization on top

I've worked in other domains my whole career, so I was astonished this week when we put a million 768-len embeddings into a vector db and it was only a few GB. Napkin math said ~25 GB and intuition said a long list of widely distributed floats would be fairly uncompressable. HNSW is pretty cool.

Re: I want everything local – Building my offline AI workspace

#240

Post by brookst »

ricardobeat wrote: post #209
PeterStuer wrote: post #180

Earlier quoted context omitted.

Of course. And I am not arguing against that at all. Just like if someone makes an inference runtime that is 4% faster, I'll take that win. But would it be the decisive factor in my choice? Only if that was my bottleneck, my true constraint. All I tried to convey was that for most of the people in the presented scenario (personal emails etc.) , a 50 or even 500GB storage requirement is not going to be that primary co…

You are glossing over the fact that for RAG you need to search over those 500GB+ which will be painfully slow and CPU-intensive. The goal is fast retrieval to add data to the LLM context. Storage space is not the sole reason to minimize the DB size.

You’re not searching over 500GB, you’re searching an index of the vectors. That’s the magic of embeddings and vector databases.

Same way you might have a 50TB relational database but “select id, name from people where country=‘uk’ and name like ‘benj%’ might only touch a few MB of storage at most.

Post reply on HN