Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

271–280 of 294 posts

Re: I want everything local – Building my offline AI workspace

#271

Post by ricardobeat »

brookst wrote: post #240
ricardobeat wrote: post #209

Earlier quoted context omitted.

You are glossing over the fact that for RAG you need to search over those 500GB+ which will be painfully slow and CPU-intensive. The goal is fast retrieval to add data to the LLM context. Storage space is not the sole reason to minimize the DB size.

You’re not searching over 500GB, you’re searching an index of the vectors. That’s the magic of embeddings and vector databases. Same way you might have a 50TB relational database but “select id, name from people where country=‘uk’ and name like ‘benj%’ might only touch a few MB of storage at most.

That’s precisely the point I tried to clear up in the previous comment.

The LEANN author proposes to create a 9GB index for a 500GB archive, and the other poster argued that it is not helpful because “storage is cheap”.

Re: I want everything local – Building my offline AI workspace

#272

Post by jychang »

cheschire wrote: post #235
oblio wrote: post #218

Earlier quoted context omitted.

Does Docker do that or are you speculating? Also - podman?

I wasn't implying docker itself was the issue. The previous commenter said that they didn't want to run a shell script that does "god knows what". The implication being that they would not trust the writer of the shell script. They wanted a docker container that would setup this offline AI workspace for them, presumably so they could interact with the AI and feed "secrets" or otherwise private data into it. Obviously…

... I mean, yes? The entire point of local AI is so you can feed your enterprise code into it, that you don't want offloaded to somewhere else.

That's the exact perfect use case for Docker, versus something heavier weight like a VM. What, you expect generic enterprise code to somehow be too dangerous for Docker but acceptable in a VM?

Re: I want everything local – Building my offline AI workspace

#273

Post by WhyNotHugo »

com2kid wrote: post #55

> Even with help from the "world's best" LLMs, things didn't go quite as smoothly as we had expected. They hallucinated steps, missed platform-specific quirks, and often left us worse off. This shows how little native app training data is even available. People rarely write blog posts about designing native apps, long winded medium tutorials don't exist, heck even the number of open source projects for native desktop…

> This shows how little native app training data is even available.

FWIW, we have very few desktop native apps nowadays. Most apps are either mobile, cli or web-based. Heck, I’m sure there’s more material online on writing cli apps than gui apps.

Re: I want everything local – Building my offline AI workspace

#274

Post by doctorpangloss »

frank_nitti wrote: post #65
doctorpangloss wrote: post #46

Earlier quoted context omitted.

The entire stack involved sends so much telemetry.

This, in particular, is a big motivator and rewarding factor in getting local setup and working. Turning off the internet and seeing everything run end to end is a joy

NVIDIA drivers send detailed telemetry.

Windows and macOS send detailed telemetry.

You have to install the pip packages and the models, which all come from websites, which collect detailed telemetry.

You don’t think Microsoft gathers detailed telemetry on all your interactions with GitHub?

The local setup doesn’t really help with that.

Re: I want everything local – Building my offline AI workspace

#275

Post by frank_nitti »

doctorpangloss wrote: post #274
frank_nitti wrote: post #65

Earlier quoted context omitted.

This, in particular, is a big motivator and rewarding factor in getting local setup and working. Turning off the internet and seeing everything run end to end is a joy

NVIDIA drivers send detailed telemetry. Windows and macOS send detailed telemetry. You have to install the pip packages and the models, which all come from websites, which collect detailed telemetry. You don’t think Microsoft gathers detailed telemetry on all your interactions with GitHub? The local setup doesn’t really help with that.

We might be talking about two different things. Yes, under normal circumstances the setup steps involve software that defaults to using telemetry -- though I'd be surprised if it's not possible anymore to achieve those in an air-gapped env using e.g. offline installers, zipped repos and wheel files, etc.

My comment was referring to runtime workloads having no telemetry (because I unplugged the internet)

Re: I want everything local – Building my offline AI workspace

#276

Post by esseph »

jauntywundrkind wrote: post #163
esseph wrote: post #147

Earlier quoted context omitted.

All of these extremely high end technologies are so far away from hitting the consumer market. Is there any desire for most people? What's the TAM?

Classic economics thinking: totally fucked "faster horses" thinking. The addressable market depends on the advantage. Which right now: we don't know. It's all a guess that someone is going to find it valuable, and no one knows. But if we find that we didn't actually need $700 NIC's to get shitty bandwidth, if we could have just been putting cables from PCIe shaped slot to PCIe slot (or oculink port!) and getting >>10…

I am very ready for optical bus lfg

Re: I want everything local – Building my offline AI workspace

#277

Post by mkagenius »

jumploops wrote: post #121

I'm a little confused about your product branding vs. blog post? From the product homepage, I imagine you're running VMs in the cloud (a la Firecracker). From the blog post though, it looks like you're running Apple-specific VMs for local execution? As someone who's built the former, I'd love the latter for use with the new gpt-oss releases :)

You are right, the product is almost the same as you described, intended for customers running LLM-generated code in their workflow.

This was something close to our hearts, so we thought of building it for our local use and releasing it for like-minded individuals.

Re: I want everything local – Building my offline AI workspace

#278

Post by zelphirkalt »

senko wrote: post #198
moffkalast wrote: post #197

Earlier quoted context omitted.

Sure but you're also constrained to only one platform. It's like the C++ vs Python argument in ML, yes writing everything in low level high speed highly optimized native code would be perfect, but ain't (almost) nobody got fucking time or skill for that.

Cross-platform toolkits are (still) a thing.

Recently I tried to make a GTK app, but the problem was, for none of the languages I tried the bindings were working well enough. So in the end I decided to make a local first static web app in Python and Django. Everything is rendered server side and state is stored in the database. If I ever finish it, it should be easy to bring it online. And then maybe registrations ...

Re: I want everything local – Building my offline AI workspace

#280

Post by akawry »

ethan_smith wrote: post #264
k__ wrote: post #212

Half-OT: Anything useful that runs reasonably fast on a regular Intel CPU/GPU?

For Intel CPUs, Phi-2 (2.7B) and TinyLlama (1.1B) run reasonably well using llama.cpp with 4-bit quantization. GGUF models with INT4 quantization typically need ~2GB RAM per billion parameters, so even older machines can handle smaller models.

Take a look at ik_llama.cpp: https://github.com/ikawrakow/ik_llama.cpp

CPU performance is much better than mainline llama, as well as having more quantization types available

Post reply on HN