Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

161–170 of 294 posts

Re: I want everything local – Building my offline AI workspace

#161

Post by jauntywundrkind »

vFunct wrote: post #143
jauntywundrkind wrote: post #78

Earlier quoted context omitted.

I agree and disagree. Many of the best models are open source, just too big to run for most people. And there are plenty of ways to fit these models! A Mac Studio M3 Ultra with 512 GB unified memory though has huge capacity, and a decent chunk of bandwidth (800GB/s. Compare vs a 5090's ~1800GB/s). $10k is a lot of money, but that ability to fit these very large models & get quality results is very impressive. Perform…

The game changer technology that'll enable full 1TB+ LLM models for cheap is Sandisk's High Bandwidth Flash. Expect devices with that in about 3-4 years, maybe even on cellphones.

I'm crazy excited for High Bandwidth Flash, really hope they pull it off. There is a huge caveat: only having a couple hundred or thousand r/w cycles before your multi $k accelerator stops working!! A pretty big constraint!

But as long as you are happy to keep running the same model, the wins here for large capacity & high bandwidth are sick ! And the affordability could be exceptional! (If you can afford to make flash with a hundred or so channels at a decent price!)

Re: I want everything local – Building my offline AI workspace

#162

Post by iezepov »

oblio wrote: post #85
andylizf wrote: post #53

This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails c…

It feels weird that the search index is bigger than the underlying data, weren't search indexes supposed to be efficient formats giving fast access to the underlying data?

Good point! Maybe indexing is a bad term here, and it's more like feature extraction (and since embeddings are high dimensional we extract a lot of features). From that point of view it makes sense that "the index" takes more space than the original data.

Re: I want everything local – Building my offline AI workspace

#163

Post by jauntywundrkind »

esseph wrote: post #147
jauntywundrkind wrote: post #128

Earlier quoted context omitted.

Thanks for the numbers. Valuable contribution for sure!! There's been a huge lag for PCIe adoption, and imo so so much has boiled down "do people need it"? In the past 10 years I feel like my eyes have been opened that every high tech company's greatest highest most compelling desire is to slow walk the release out. To move as slow as the market will bear, to do as little as possible, to roll on and on with minor inc…

All of these extremely high end technologies are so far away from hitting the consumer market. Is there any desire for most people? What's the TAM?

Classic economics thinking: totally fucked "faster horses" thinking.

The addressable market depends on the advantage. Which right now: we don't know. It's all a guess that someone is going to find it valuable, and no one knows.

But if we find that we didn't actually need $700 NIC's to get shitty bandwidth, if we could have just been putting cables from PCIe shaped slot to PCIe slot (or oculink port!) and getting >>10x performance with >>10x less latency? Yeah bro uhh I think there might be a desire for using the same fucking chip we already use but getting 10x + 10x better out of it.

Faster lower latency cheaper storage? RAM expandability? Lower latency GPU access? There's so much that could make a huge difference for computing, broadly.

Re: I want everything local – Building my offline AI workspace

#164

Post by vgb2k18 »

Imustaskforhelp wrote: post #35

I think I still prefer local but I feel like that's because that most AI inference is kinda slow or comparable to local. But I recently tried out cerebras or (I have heard about groq too) and honestly when you try things at 1000 tk/s or similar, your mental model really shifts and becomes quite impatient. Cerebras does say that they don't log your data or anything in general and you would have to trust me to say that…

> Cerebras does say that they don't log your data or anything in general

Unil a judge says they must log everything, indefinitely

Re: I want everything local – Building my offline AI workspace

#165

Post by catlifeonmars »

iezepov wrote: post #162
oblio wrote: post #85

Earlier quoted context omitted.

It feels weird that the search index is bigger than the underlying data, weren't search indexes supposed to be efficient formats giving fast access to the underlying data?

Good point! Maybe indexing is a bad term here, and it's more like feature extraction (and since embeddings are high dimensional we extract a lot of features). From that point of view it makes sense that "the index" takes more space than the original data.

Why would the embeddings be higher dimensionally than the data? I imagine the embeddings would contain relatively higher entropy (and thus lower redundancy) than many types of source data.

Re: I want everything local – Building my offline AI workspace

#166

Post by jychang »

Any way to install this via just a container?

Similar to a `docker compose up -d` that a lot of projects offer. Just download the docker-compose.yml file into a folder, run the command, and you're running. If you want to delete everything, just `docker compose down` and delete the folder, and the container and everything is gone.

Anything similar to that? I don't want to run a random install.sh on my machine that does god knows what.

Re: I want everything local – Building my offline AI workspace

#167

Post by motorest »

ac29 wrote: post #113
motorest wrote: post #94

Earlier quoted context omitted.

> Fast-forward some number of years (...) I repeat: OP's home server costs as much as a few months of a cloud provider's infrastructure. To put it another way, OP can buy brand new hardware a few times per year and still save money compared with paying a cloud provider for equivalent hardware. > Regarding the equivalent EC2 instance, I'm not comparing it to the cost of a homelab, I'm comparing it to the cost of an An…

> OP stated quite clearly their goal was to run models locally. Fair, but at the point you trust Amazon hosting your "local" LLM, its not a huge reach to just use Amazon Bedrock or something

> Fair, but at the point you trust Amazon hosting your "local" LLM, its not a huge reach to just use Amazon Bedrock or something

I don't think you even bothered to look at Amazon Bedrock's pricing before doing that suggestion. They charge users per input tokens + output tokens. In Amazon Bedrock, a single chat session involving 100k tokens can cost you $200. That alone is a third of OP's total infrastructure costs.

If you want to discuss options in terms of cost, the very least you should do is look at pricing.

Re: I want everything local – Building my offline AI workspace

#168

Post by NJL3000 »

sebmellen wrote: post #56
andylizf wrote: post #53

This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails c…

I know next to nothing about embeddings. Are there projects that implement this same “pruned graph” approach for cloud embeddings?

It’s in the works… been meaning to do a show HN moment to see if it flies or I Fall on my face..

Re: I want everything local – Building my offline AI workspace

#169

Post by spauldo »

thorncorona wrote: post #92
com2kid wrote: post #55

> Even with help from the "world's best" LLMs, things didn't go quite as smoothly as we had expected. They hallucinated steps, missed platform-specific quirks, and often left us worse off. This shows how little native app training data is even available. People rarely write blog posts about designing native apps, long winded medium tutorials don't exist, heck even the number of open source projects for native desktop…

I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?

A lot of us just don't want to be web developers. I mostly write IEC 61131 code, with sprinkles of BASIC (yuck), C, Perl, and Lisp. I've used JavaScript and quite frankly, you can keep it.
Post reply on HN