Earlier quoted context omitted.
Out of curiosity have you tried other small models? The e2b for me was unusable. Llama3.2 3b was better and that thing is a year old and I rarely use it now too.
yes i keep on trying small models, i have also tried qwen 3.5 0.8B, 2B, 4b and gemma4 e4B models but they either did not worked reliably (thinking loop, issue in following instruction) or there were performance issues (prompt speed, tg speed, too much ram) e2b was the sweet spot where i could give it plan and it can edit files properly.
Running local models on an M4 with 24GB memory
131–140 of 189 posts
Re: Running local models on an M4 with 24GB memory
#132I understand that multiple things can be true at the same time. Is the concern for centralized AI monopolization? Or is the concern for the art of software engineering?
Re: Running local models on an M4 with 24GB memory
#133I don’t understand the bipolar nature on hacker news towards LLMs. On the one hand, they’re destroying the art of software development and we shouldn’t use them. But on the other hand, there’s a lot of excitement around running them locally. I understand that multiple things can be true at the same time. Is the concern for centralized AI monopolization? Or is the concern for the art of software engineering?
Re: Running local models on an M4 with 24GB memory
#134Re: Running local models on an M4 with 24GB memory
#135I think it's useful to be realistic about what you can do with a local model, especially something as small as the 9B the author is using. A 9B model is around the level of Sonnet 3.6 - it can do autocomplete and small functions but it loses track trying to understand large problems. But the are interesting and fun to play with! I do a LOT of work on local agent harnesses etc, mostly for fun. My current project is a…
But I was doing a lot more than autocomplete and small functions with Sonnet 3.5.
Re: Running local models on an M4 with 24GB memory
#136It seems like cache layers like https://omlx.ai make more RAM better than more GPU cores or faster CPUs cores, but I'm curious if someone has tested both.
Re: Running local models on an M4 with 24GB memory
#137I don’t understand the bipolar nature on hacker news towards LLMs. On the one hand, they’re destroying the art of software development and we shouldn’t use them. But on the other hand, there’s a lot of excitement around running them locally. I understand that multiple things can be true at the same time. Is the concern for centralized AI monopolization? Or is the concern for the art of software engineering?
There are actually multiple people here. It's not just one person with many accounts...
Re: Running local models on an M4 with 24GB memory
#138Earlier quoted context omitted.
But I was doing a lot more than autocomplete and small functions with Sonnet 3.5.
I agree, earlier Sonnet wasn't that great, but Sonnet 3.5 is where things really came together. The difference was night-and-day. Sonnet 3.7, 4.0, 4.5, etc... didn't have as drastic of a change to me.
Re: Running local models on an M4 with 24GB memory
#139In my experience (so far), I can’t let the LLM write too much in one go.
I need to test the hell out of what it gives me, and I can’t ask for too much, at one time.
I tend to ask it to “flesh out” functions, where I have a signature, and a detailed headerdoc comment. I will provide a lot of guidance about the context, often attaching relevant files.
Even then, it often doesn’t give me what I need, first time, unless it’s a small function, with extremely limited scope.
That said, it’s been extremely helpful. It has accelerated my development greatly.
I have found that it gives me much better PHP, than Swift.
I suspect that may be because PHP is extremely mature, and there’s millions and millions of lines of high-quality code out there, in open-source repos, while Swift is probably mostly in closed repos, with open stuff not really provided by experienced developers (it’s a proprietary language used for shipping commercial software, so that may also apply to other languages).
What it gives me in Swift, most closely resembles stuff that enthusiastic newer folks would do, and want to show off.
Re: Running local models on an M4 with 24GB memory
#140Earlier quoted context omitted.
This. I have seen way too many people who are overly optimistic about local LLMs. Having spent a decent amount of time playing with them on consumer nvidia GPUs, I understand well that they not going to be widely usable any time soon. Unfortunately not many people share that.
So the cofounder of hugging face made a post about qwen 3.6 being atclaude level of performance for the lols? When were you trying local models? The model releases from April 2026 are a serious change in performance.
These are so far from Opus it's not even funny. They are not close to being in the same league. Gemma might be like a frontier model from a couple years ago, but with much worse performance in long context chats.