Live data from Hacker News

Running local models on an M4 with 24GB memory

jola.dev

131–140 of 189 posts

Re: Running local models on an M4 with 24GB memory

#131
post #127

Earlier quoted context omitted.

Out of curiosity have you tried other small models? The e2b for me was unusable. Llama3.2 3b was better and that thing is a year old and I rarely use it now too.

yes i keep on trying small models, i have also tried qwen 3.5 0.8B, 2B, 4b and gemma4 e4B models but they either did not worked reliably (thinking loop, issue in following instruction) or there were performance issues (prompt speed, tg speed, too much ram) e2b was the sweet spot where i could give it plan and it can edit files properly.

That makes sense it sounds like your computer isn't super powerful. Whatever works for you

Re: Running local models on an M4 with 24GB memory

#132
I don’t understand the bipolar nature on hacker news towards LLMs. On the one hand, they’re destroying the art of software development and we shouldn’t use them. But on the other hand, there’s a lot of excitement around running them locally.

I understand that multiple things can be true at the same time. Is the concern for centralized AI monopolization? Or is the concern for the art of software engineering?

Re: Running local models on an M4 with 24GB memory

#133

I don’t understand the bipolar nature on hacker news towards LLMs. On the one hand, they’re destroying the art of software development and we shouldn’t use them. But on the other hand, there’s a lot of excitement around running them locally. I understand that multiple things can be true at the same time. Is the concern for centralized AI monopolization? Or is the concern for the art of software engineering?

hn is not a single person, there are a variety of people here with a variety of experiences and opinions and biases.

Re: Running local models on an M4 with 24GB memory

#135
post #21

I think it's useful to be realistic about what you can do with a local model, especially something as small as the 9B the author is using. A 9B model is around the level of Sonnet 3.6 - it can do autocomplete and small functions but it loses track trying to understand large problems. But the are interesting and fun to play with! I do a LOT of work on local agent harnesses etc, mostly for fun. My current project is a…

But I was doing a lot more than autocomplete and small functions with Sonnet 3.5.

I agree, earlier Sonnet wasn't that great, but Sonnet 3.5 is where things really came together. The difference was night-and-day. Sonnet 3.7, 4.0, 4.5, etc... didn't have as drastic of a change to me.

Re: Running local models on an M4 with 24GB memory

#136
Is it better to have an M4-M5 Pro with 32GB of ram or an M1-M2 Max with 64GB of ram? They seem about the same price.

It seems like cache layers like https://omlx.ai make more RAM better than more GPU cores or faster CPUs cores, but I'm curious if someone has tested both.

Re: Running local models on an M4 with 24GB memory

#137

I don’t understand the bipolar nature on hacker news towards LLMs. On the one hand, they’re destroying the art of software development and we shouldn’t use them. But on the other hand, there’s a lot of excitement around running them locally. I understand that multiple things can be true at the same time. Is the concern for centralized AI monopolization? Or is the concern for the art of software engineering?

>I don’t understand the bipolar nature on hacker news towards LLMs.

There are actually multiple people here. It's not just one person with many accounts...

Re: Running local models on an M4 with 24GB memory

#138

Earlier quoted context omitted.

But I was doing a lot more than autocomplete and small functions with Sonnet 3.5.

I agree, earlier Sonnet wasn't that great, but Sonnet 3.5 is where things really came together. The difference was night-and-day. Sonnet 3.7, 4.0, 4.5, etc... didn't have as drastic of a change to me.

I remember even after 3.7 was released I kept using 3.5 in Cursor because it just did exactly what I wanted

Re: Running local models on an M4 with 24GB memory

#139
> The longer you let it drive without constraints, the worse the wreckage gets. The velocity makes you think you're winning right up until the moment everything collapses simultaneously.

In my experience (so far), I can’t let the LLM write too much in one go.

I need to test the hell out of what it gives me, and I can’t ask for too much, at one time.

I tend to ask it to “flesh out” functions, where I have a signature, and a detailed headerdoc comment. I will provide a lot of guidance about the context, often attaching relevant files.

Even then, it often doesn’t give me what I need, first time, unless it’s a small function, with extremely limited scope.

That said, it’s been extremely helpful. It has accelerated my development greatly.

I have found that it gives me much better PHP, than Swift.

I suspect that may be because PHP is extremely mature, and there’s millions and millions of lines of high-quality code out there, in open-source repos, while Swift is probably mostly in closed repos, with open stuff not really provided by experienced developers (it’s a proprietary language used for shipping commercial software, so that may also apply to other languages).

What it gives me in Swift, most closely resembles stuff that enthusiastic newer folks would do, and want to show off.

Re: Running local models on an M4 with 24GB memory

#140
post #68

Earlier quoted context omitted.

This. I have seen way too many people who are overly optimistic about local LLMs. Having spent a decent amount of time playing with them on consumer nvidia GPUs, I understand well that they not going to be widely usable any time soon. Unfortunately not many people share that.

So the cofounder of hugging face made a post about qwen 3.6 being atclaude level of performance for the lols? When were you trying local models? The model releases from April 2026 are a serious change in performance.

It's just not there yet. I have tried all the models from April, including the Gemma 4 variants.

These are so far from Opus it's not even funny. They are not close to being in the same league. Gemma might be like a frontier model from a couple years ago, but with much worse performance in long context chats.

Post reply on HN