Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

251–260 of 294 posts

Re: I want everything local – Building my offline AI workspace

#251

Post by flanger »

zozbot234 wrote: post #231
t_mann wrote: post #224

Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great f…

What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project. Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.

The founders of Exo ghosted the dev community and went closed-source. Nobody has heard from them. I wish people would stop recommending Exo (a tribute to their marketing) and check out GPUStack instead. Overall another rug pull by the devs as soon as they got traction.

Re: I want everything local – Building my offline AI workspace

#252

Post by Aurornis »

com2kid wrote: post #55

> Even with help from the "world's best" LLMs, things didn't go quite as smoothly as we had expected. They hallucinated steps, missed platform-specific quirks, and often left us worse off. This shows how little native app training data is even available. People rarely write blog posts about designing native apps, long winded medium tutorials don't exist, heck even the number of open source projects for native desktop…

You left out the next lines, which add some important context:

> Then we tried wrapping a NextJS app inside Electron. It took us longer than we'd like to admit. As of this writing, it looks like there's just no (clean) way to do it.

> So, we gave up on the Mac app.

They weren't writing a fully native app. They started with a NextJS web app and then tried to put it inside Electron, a cross-platform toolkit.

All the training data in the world about native app development wouldn't have helped here. They were using a recent JS framework and trying to put it in a relatively recent cross-platform tool. The two parts weren't made to work together so training data likely doesn't exist, other than maybe some small amount of code or issues on GitHub discussing problems with the approach.

Re: I want everything local – Building my offline AI workspace

#253

Post by pbronez »

zozbot234 wrote: post #231
t_mann wrote: post #224

Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great f…

What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project. Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.

Jeff Geerling just did a video with a cluster of 4 Framework Desktop main boards. He put a decent amount of work into Exo and concluded it’s a VC Rugpull… abandoned as soon as it won some attention.

He also explored several other open source AI scale out libraries, and reported that they’re generally way less mature than tooling for traditional scientific cluster computing.

https://www.jeffgeerling.com/blog/2025/i-clustered-four-fram...

Re: I want everything local – Building my offline AI workspace

#254

Post by zozbot234 »

flanger wrote: post #251
zozbot234 wrote: post #231

Earlier quoted context omitted.

What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project. Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.

The founders of Exo ghosted the dev community and went closed-source. Nobody has heard from them. I wish people would stop recommending Exo (a tribute to their marketing) and check out GPUStack instead. Overall another rug pull by the devs as soon as they got traction.

Why can't that dev community just fork the project under a new name and maintain it properly? Picking up a third-party project is absolutely par for the course in FLOSS development.

Re: I want everything local – Building my offline AI workspace

#255

Post by mockingloris »

shekhargulati wrote: post #211

I tried to port it to Docker and wrote a blog here https://shekhargulati.com/2025/08/09/making-coderunner-ui-wo... . I used Claude Code to do the port. We used Datalayer Jupyter MCP Server instead of coderunner which uses Apple containers.

At least, you are honest about augmenting the porting process. It's amazing what one can accomplish when they realize that with proper time, planning and a good grounding on building code/systems, that a lot more is possible.

The takeaway for me is that because these tools are fast doesn't mean the task also needs to move as fast. At least till AGI, a sound human reasoning before hitting enter goes a long way.

Thanks for sharing

Re: I want everything local – Building my offline AI workspace

#257

Post by fouc »

zozbot234 wrote: post #231
t_mann wrote: post #224

Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great f…

What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project. Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.

There's a couple of alternatives to exo it seems https://github.com/b4rtaz/distributed-llama and https://github.com/ray-project/ray

Re: I want everything local – Building my offline AI workspace

#258

Post by m-s-y »

zozbot234 wrote: post #231
t_mann wrote: post #224

Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great f…

What's the deal with Exo anyway? I've seen it described as an abandoned, unmaintained project. Anyway, you don't really need a lot of fast RAM unless you insist on getting a real-time usable response. If you're fine with running a "good" model overnight or thereabouts, there are things you can do to get better use of fairly low-end hardware.

It’s functional if your goal is to run models that won’t fit into RAM on a single machine. Functional.

the slow interconnects (yes, even at 40Gbps thunderbolt) severely limit both TtFT and tokens/second.

I tried it extensively for a few days, and ended up getting a single M3 Ultra Mac Studio, and am loving life.

Re: I want everything local – Building my offline AI workspace

#259

Post by __MatrixMan__ »

cheschire wrote: post #206
jychang wrote: post #166

Any way to install this via just a container? Similar to a `docker compose up -d` that a lot of projects offer. Just download the docker-compose.yml file into a folder, run the command, and you're running. If you want to delete everything, just `docker compose down` and delete the folder, and the container and everything is gone. Anything similar to that? I don't want to run a random install.sh on my machine that doe…

But you would pump your secrets into a docker AI?

If it was sufficiently locked down, yeah. It's only going to live long enough to give me an answer and then everything it can write to goes away afterwards (besides the answer itself).

What harm can it do?

Re: I want everything local – Building my offline AI workspace

#260

Post by m-s-y »

api wrote: post #247
t_mann wrote: post #224

Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great f…

Prices are still coming down. Assuming that keeps happening we will have laptops with enough RAM in the sub-2k range in 5 years. Question is whether models will keep getting bigger. If useful model sizes plateau eventually a good model becomes something at least many people can easily run locally. If models keep usefully growing this doesn’t happen. The largest ones I see are in the 405g range which quantized fits in…

GPUs are already effectively ASICs for the math that runs both 3D scenes and LLMs, no?
Post reply on HN