Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models on an M4 with 24GB memory

jola.dev

Post reply on HN
Print view View original

91–100 of 189 posts

Re: Running local models on an M4 with 24GB memory

#91

Post by zozbot234 »

isomorphic wrote: post #80
discordance wrote: post #58

Earlier quoted context omitted.

Could you please share your time to first token and tok/s?

M4 Pro 64GB (14 CPU / 20 GPU), Gemma 4 31B Q4_K_M GGUF, LM Studio: time to first token 0.92s, 11.56 tokens/s. Edit: For comparison with the other poster, same setup as above, but with Gemma 4 31B Instruct 8bit MLX (not sure if exactly the same model): time to first token 4.62s, 7.20 tokens/s; with a different prompt, 1.17s and 7.24 tokens/s.

Could you (or anyone with the same hardware) try antirez's ds4 and report how gracefully it degrades with only the 64GB RAM? Obviously it's going to be dog slow at best for any single inference flow, but can you meaningfully improve on that by running many sessions in parallel? (Ideally you'd need roughly on the order of model sparsity in order to get meaningful sharing of MoE weights, but whether that's genuinely achievable is anyone's guess!)

Re: Running local models on an M4 with 24GB memory

#92

Post by PAndreew »

Critics are (rightly) pointing to the fact that these models are not on par with SOTA for complex coding tasks. But many seems to forget that a large part of white collar office work is Excel crushing, file moving, translating dry legal documents, e-mail drafting, PPT drudgery, etc. These are absolutely doable with 30-35b+ models with the added benefit of keeping company data private.

Re: Running local models on an M4 with 24GB memory

#94

Post by tjoff »

PAndreew wrote: post #92

Critics are (rightly) pointing to the fact that these models are not on par with SOTA for complex coding tasks. But many seems to forget that a large part of white collar office work is Excel crushing, file moving, translating dry legal documents, e-mail drafting, PPT drudgery, etc. These are absolutely doable with 30-35b+ models with the added benefit of keeping company data private.

Arguably excel and legal are much worse than code because catching the mistakes can be much harder.

Case in point, JPMorgan London Whale incident, $6 billion loss caused by an excel error...

Re: Running local models on an M4 with 24GB memory

#95

Post by darkstar_16 »

HDBaseT wrote: post #56
solenoid0937 wrote: post #41

Earlier quoted context omitted.

> It is absolutely not comparable to frontier models. This is not said often enough. Yes, local LLMs are great! But reading most HN posts on the subject, you'd think they're within reach of Opus 4.7. There is a very small, very vocal, very passionate crowd that dramatically overstates the capabilities of local LLMs on HN.

At least in my experience, local models are very far away from models like Opus 4.7 or ChatGPT 5.5 in coding and problem solving areas. I find them useful in basic research and learning and question asking tasks. Although at the same time, a Wikipedia page read or a few Google searches likely could accomplish the same and has been able to for decades.

I think you're doing it wrong. Use the frontier moddels for the research, planning etc and once you have a plan give it to a local model for implementation.

Re: Running local models on an M4 with 24GB memory

#96

Post by ActorNightly »

solenoid0937 wrote: post #41
quacker wrote: post #37

I could have used this article before I spent the weekend arriving to the same conclusion! Same laptop, and my contrived test was having it fix 50 or so lint errors in a small vibe-coded C++ repo. I wanted it to be able to handle a bunch of small tasks without getting stuck too often. GPT OSS 20B was usable but slow, and actually frequently made mistakes like adding or duplicating statements unnecessarily, listing th…

> It is absolutely not comparable to frontier models. This is not said often enough. Yes, local LLMs are great! But reading most HN posts on the subject, you'd think they're within reach of Opus 4.7. There is a very small, very vocal, very passionate crowd that dramatically overstates the capabilities of local LLMs on HN.

Im like 50% convinced that these people are paid by Apple to promote their products. Because the conversation is always just being able to execute models (even larger ones), on mac hardware with unified memory, but nobody ever mentions that inference speed is unusably slow.

You can have good local LLM performance through agents, but you need fast inference. Generally, 2x 3090 or at the minimum 2x3080s (you need 2 to speed up prefill processing to build KV Cache). You just ironically need to be good at prompt engineering, which has a lot of analogue in real world on being able to manage low skilled people in completing tasks.

Re: Running local models on an M4 with 24GB memory

#97

Post by PAndreew »

tjoff wrote: post #94
PAndreew wrote: post #92

Critics are (rightly) pointing to the fact that these models are not on par with SOTA for complex coding tasks. But many seems to forget that a large part of white collar office work is Excel crushing, file moving, translating dry legal documents, e-mail drafting, PPT drudgery, etc. These are absolutely doable with 30-35b+ models with the added benefit of keeping company data private.

Arguably excel and legal are much worse than code because catching the mistakes can be much harder. Case in point, JPMorgan London Whale incident, $6 billion loss caused by an excel error...

Yes... I mean organisations have to adapt to this new working scheme. First they need new processes (maybe borrowed from SW development) that enables them to triage work products on a risk/reward scale. For example my wife works on medical device tenders. It is obligatory to translate every frikkin Word document to our native language which in the end noone will read. Do we use LLMs to do the translation? Hell yeah. For a critical legal document? Eeee. Also I think enablers like speical harnesses shall be developed/improved by keeping these folks in mind. For example to build hooks into the harness that forces the LLM to test/review/sample its output. So yes it's a complex topic, but my point was rather that the inherent capabilities of medium-large-ish open LLMs are sufficient for let's say 70-80% of such office work, and it's a huge market.

Re: Running local models on an M4 with 24GB memory

#98

Post by adornKey »

reillyse wrote: post #36

so, interested how many people are running higher end AI models locally? Figure if I'm spending $800/month on tokens I can build a pretty beefy local machine for the cost of a few months spend - what is people's experience with say a $5k server custom built (and only for) running an AI model.

I'm running a server in the 5K-league. And the results are very good. I get about 150 Tokens/s from Qwen3 for coding. And about 50 Tokens/s from the newer non-MoE Qwens.

I wouldn't bother with less than 32GB of VRAM. With 16GB you can already run something usable, but 32GB gives you much more power. 9B and 14B are only interesting if you want to tune models yourself. The sweet spot now seem to be around 27B-35B.

Re: Running local models on an M4 with 24GB memory

#99

Post by hparadiz »

thot_experiment wrote: post #78
thot_experiment wrote: post #62

Earlier quoted context omitted.

Gemma 4 IS good, I've literally had it get a thing right that Opus 4.7 missed, the edges are ragged and I'm reliably finding usecases where it's basically equivalent. Ultimately the metric is "what can I RELY on it to do". Opus definitely knows a lot more and can sometimes do much more complex tasks, but especially when you're good about feeding the context Gemma is amazing. The difference between the sets of things…

Re-posting this from a buried comment for visibility because it's just so fucking impressive to me. I went to the store to buy mixers and while I was out Gemma 4 31b got pretty far along with reverse engineering the bluetooth protocol of a desk thermometer I have. I forgot to turn on the web search tool, so it just went at it, writing more and more specific diagnostic logging/probing tools over the course of like 8 t…

A technology indistinguishable from magic.

Re: Running local models on an M4 with 24GB memory

#100

Post by ThomasBb »

brrrrrm wrote: post #70
ThomasBb wrote: post #67

Beyond the models getting better; there are still huge gains available in the inference engine side with new tricks like Dflash, MRT, turboquant - for some usecases these can multiply the speeds. There are even some model specific optimized kernels like for DeepSeek 4 flash that seem wild. Makes me feel we are nowhere near the optimum yet. Examples: https://dasroot.net/posts/2026/05/gemma-4-speed-hacks-mtp-df... http…

what's MRT?

Sorry, autocorrect got me there: MTP is what I meant.
Post reply on HN