Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

131–140 of 294 posts

Re: I want everything local – Building my offline AI workspace

#131

Post by lexh »

ac29 wrote: post #115
c-hendricks wrote: post #17

Earlier quoted context omitted.

Why would AI be one of the few areas where locally-hosted options can't reach "good enough"?

Maybe a better question is when will SOTA models be "good enough"? At the moment there appears to be ~no demand for older models, even models that people praised just a few months ago. I suspect until AGI/ASI is reached or progress plateaus, that will continue be the case.

The current SOTA closed model providers are also all rolling out access to their latest models with better pricing (e.g. GPT-5 this week), which seems like a confounding factor unique to this moment in the cycle. An API consumer would need to have a very specific reason to choose GPT-4o over GPT-5, given the latter costs less, benchmarks better and is roughly the same speed.

Re: I want everything local – Building my offline AI workspace

#132

Post by ekianjo »

tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

once the models behind API start monetization of their results, their outputs will get much worse. Its just a matter of time.

Re: I want everything local – Building my offline AI workspace

#133

Post by moron4hire »

bionsystem wrote: post #84
1oooqooq wrote: post #76

Earlier quoted context omitted.

more interesting is the extent apple convinced people a laptop can replace a desktop or server. mind blowing reality distortion field (as will be proven by some twenty comments telling I'm wrong 3... 2... 1).

I'm a desktop guy, considering the switch to a laptop-only setup, what would I miss ?

You'll end up with a portable desktop with bad thermals, impacting performance, battery life, and actually-on-the-lap comfort. Bleeding-edge performance laptops can really only manage an hour, max, on battery, making the form factor much more about moving between different pre-planned, desk-oriented work locations.

I take my laptop back and forth from home to work. At work, I ban them from in-person meetings because I want people to actually pay attention to the meeting. In both locations where I use the computer, I have a monitor, keyboard, and mouse I'm plugging in via a dock. That makes the built-in battery and I/O redundant. I think I would rather have a lower-powered, high-battery, ultra portable laptop remoting into the desktop for the few times I bring my computer to in-person meetings for demos.

I wish the memory bandwidth for eGPUs was better.

Re: I want everything local – Building my offline AI workspace

#134

Post by jarym »

Playing with local LLMs is indeed fun. I use Kasm workspaces[0] to run a desktop session with ollama running on the host. Gives me the isolation and lets me experiment with all manner of crazy things (I tried to make a computer-use AI but it wasn't very good)

[0] https://kasmweb.com/

Re: I want everything local – Building my offline AI workspace

#135

Post by aldanor »

moron4hire wrote: post #133
bionsystem wrote: post #84

Earlier quoted context omitted.

I'm a desktop guy, considering the switch to a laptop-only setup, what would I miss ?

You'll end up with a portable desktop with bad thermals, impacting performance, battery life, and actually-on-the-lap comfort. Bleeding-edge performance laptops can really only manage an hour, max, on battery, making the form factor much more about moving between different pre-planned, desk-oriented work locations. I take my laptop back and forth from home to work. At work, I ban them from in-person meetings because…

Huh? Bleeding edge laptops can last a lot more on battery. M3 16'' mbp lasts definitely enough for a full office day of coding. Twice that if just browsing and not doing cpu intensive stuff.

Re: I want everything local – Building my offline AI workspace

#136

Post by fredmcawesome »

tcdent wrote: post #103
motorest wrote: post #61

Earlier quoted context omitted.

You're using a lot of words to say "I believe yesterday's hardware might not run models as as fast as today's hardware." That's fine. The point is that yesterday's hardware is quite capable of running yesterday's models, and obviously it will also run tomorrow's models. So the question is cost. Capex vs opex. The fact is that buying your own hardware is proven to be far more cost-effective than paying cloud providers…

I incorporated the quantization aspect because it's not that simple. Yes, old hardware will be slower, but you will also need a significant amount more of it to even operate. RAM is the expensive part. You need lots of it. You need even more of it for older hardware which has less efficient float implementations. https://developer.nvidia.com/blog/floating-point-8-an-introd...

But surely this is short term? Once you get older hardware with FP4 support this shouldn't be a concern.

Re: I want everything local – Building my offline AI workspace

#137

Post by wizee »

Aurornis wrote: post #47
tcdent wrote: post #11

I'm constantly tempted by the idealism of this experience, but when you factor in the performance of the models you have access to, and the cost of running them on-demand in a cloud, it's really just a fun hobby instead of a viable strategy to benefit your life. As the hardware continues to iterate at a rapid pace, anything you pick up second-hand will still deprecate at that pace, making any real investment in hardw…

I think the local LLM scene is very fun and I enjoy following what people do. However every time I run local models on my MacBook Pro with a ton of RAM, I’m reminded of the gap between local hosted models and the frontier models that I can get for $20/month or nominal price per token from different providers. The difference in speed and quality is massive. The current local models are very impressive, but they’re sti…

While cloud models are of course faster and smarter, I've been pretty happy running Qwen 3 Coder 30B-A3B on my M4 Max MacBook Pro. It has been a pretty good coding assistant for me with Aider, and it's also great for throwing code at and asking questions. For coding specifically, it feels roughly on par with SOTA models from mid-late 2024.

At small contexts with llama.cpp on my M4 Max, I get 90+ tokens/sec generation and 800+ tokens/sec prompt processing. Even at large contexts like 50k tokens, I still get fairly usable speeds (22 tok/s generation).

Re: I want everything local – Building my offline AI workspace

#138

Post by davidmurdoch »

1oooqooq wrote: post #76
Aurornis wrote: post #47

Earlier quoted context omitted.

I think the local LLM scene is very fun and I enjoy following what people do. However every time I run local models on my MacBook Pro with a ton of RAM, I’m reminded of the gap between local hosted models and the frontier models that I can get for $20/month or nominal price per token from different providers. The difference in speed and quality is massive. The current local models are very impressive, but they’re sti…

more interesting is the extent apple convinced people a laptop can replace a desktop or server. mind blowing reality distortion field (as will be proven by some twenty comments telling I'm wrong 3... 2... 1).

I dropped $4k on an (Intel) laptop a few years ago. I thought it would blow my old 2012 core i7 out of the water. Editing photos in Lightroom and Photoshop often requires heavy sustained CPU work. Thermals in laptops is just not a solved problem. People who say laptops are fine replacements for desktops probably don't realize how much and how quickly thermals limit heavy multi-core CPU workloads.

Re: I want everything local – Building my offline AI workspace

#139

Post by moron4hire »

aldanor wrote: post #135
moron4hire wrote: post #133

Earlier quoted context omitted.

You'll end up with a portable desktop with bad thermals, impacting performance, battery life, and actually-on-the-lap comfort. Bleeding-edge performance laptops can really only manage an hour, max, on battery, making the form factor much more about moving between different pre-planned, desk-oriented work locations. I take my laptop back and forth from home to work. At work, I ban them from in-person meetings because…

Huh? Bleeding edge laptops can last a lot more on battery. M3 16'' mbp lasts definitely enough for a full office day of coding. Twice that if just browsing and not doing cpu intensive stuff.

Even the M4 Max is not "bleeding edge". Apple is doing impressive stuff with energy efficient compute, but you can't get top of the line raw compute for any amount of financial of energy budget from them.

Re: I want everything local – Building my offline AI workspace

#140

Post by wizee »

Uehreka wrote: post #70
Aurornis wrote: post #47

Earlier quoted context omitted.

I think the local LLM scene is very fun and I enjoy following what people do. However every time I run local models on my MacBook Pro with a ton of RAM, I’m reminded of the gap between local hosted models and the frontier models that I can get for $20/month or nominal price per token from different providers. The difference in speed and quality is massive. The current local models are very impressive, but they’re sti…

I was talking about this in another comment, and I think the big issue at the moment is that a lot of the local models seem to really struggle with tool calling. Like, just straight up can’t do it even though they’re advertised as being able to. Most of the models I’ve tried with Goose (models which say they can do tool calls) will respond to my questions about a codebase with “I don’t have any ability to read files,…

Qwen 3 Coder 30B-A3B has been pretty good for me with tool calling.
Post reply on HN