My local model setup on an M4 Pro Mac Mini
1–10 of 208 posts
Re: My local model setup on an M4 Pro Mac Mini
#2No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
Re: My local model setup on an M4 Pro Mac Mini
#3No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.
Re: My local model setup on an M4 Pro Mac Mini
#4No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.
Re: My local model setup on an M4 Pro Mac Mini
#5No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minutes to run. It's free. The machine is just sitting there anyway. I have tried using qwen-coder and opencode on my M5 Max 128gb and compared to claude code it's painful. I did setup a workflow where claude plans, qwen executes (unattended overnight, again b/c it's slow) and then claude reviews. I benchmarked this several times and I ended up using MORE tokens with claude because it had to 'fix' all the qwen issues. While the code it produced was 'good enough' the fixes were worth it so I just stick to coding task using API models (codex and claude).
Re: My local model setup on an M4 Pro Mac Mini
#6yes, share performance, numbers if you can, also i wonder if you figured out a way to do a 2way audio with local models, or even explored that. I have a very similar setup but not too happy with the token speed, will try omlx though !!!
Re: My local model setup on an M4 Pro Mac Mini
#7I have an m1 Mac 64gb and look forward to trying this out
Not many people share setup with actual setup handholding so that was very G of you
Re: My local model setup on an M4 Pro Mac Mini
#8I tried the 1 bit model of Qwen3.6 27B on my M1 pro (16G) and got 13 tok/s with only 5G of ram usage.
https://x.com/mkagenius/status/2093730391429685732
(xcancel seems to have received a cease and desist)
Re: My local model setup on an M4 Pro Mac Mini
#9My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency.
I've since acquired two DGX Sparks, and it feels so much snappier.
Re: My local model setup on an M4 Pro Mac Mini
#10Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense model
I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me)
Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.