Been curious about the M4 Pro for local models. My M3 Max machine still chugs on larger LLMs, definitely eyeing an upgrade.
My local model setup on an M4 Pro Mac Mini
81–90 of 207 posts
Re: My local model setup on an M4 Pro Mac Mini
#82Similar setup here with an M3 Max, it's surprisingly capable. Curious what models you're running on that M4 Pro.
Re: My local model setup on an M4 Pro Mac Mini
#83No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
> I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible
With all due respect, I'm not clear why you are so surprised ?
By your own admission its a little mini-PC with 16GB RAM, I'm not sure what miracles you were expecting ?
Its a bit like complaining Rasperry Pi performance is terrible when trying to compile the Linux kernel.
Re: My local model setup on an M4 Pro Mac Mini
#84> The main reason to run local: cloud APIs are rented land. They can change their pricing, hit your usage limits, or swap the model being served behind the scenes whenever they feel like it.
Yes but it's easy to replace them.
The main reason should be privacy.
Re: My local model setup on an M4 Pro Mac Mini
#85Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that? If there was a "Mullvad of GPU clouds", would that solve…
yes, and it's already some offerings like that but they all cost a lot because they only good for "I have some idea of workload for N hours or days" lets rent it and run. That fine for some experimentation but if you think about renting something 24/7 even for example to share it with the friends that will cost at least 4x from any API prices as result (something like rtx 6000 48gb will cost ~$470/m).
Re: My local model setup on an M4 Pro Mac Mini
#86No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…
> It's free
It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.
Re: My local model setup on an M4 Pro Mac Mini
#87[flagged]
Re: My local model setup on an M4 Pro Mac Mini
#88[flagged]
Re: My local model setup on an M4 Pro Mac Mini
#89No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?
Qwen3.8:27b-mlx on 64GB MBP M4, I can get up to 42tok/s, more often than not in the ~30 range.
Re: My local model setup on an M4 Pro Mac Mini
#90Earlier quoted context omitted.
I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…
> It's free It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.
> It's very possible that it costs you more than a cloud mode would
...which is almost always true in a single request/reply mode and never true in batch mode. Single request usually 2x-3x more expensive than cloud and batch mode 2x-3x cheaper. Now, for narrow tasks, a finetuned tiny 8b model would dramatically outperform SOTA frontiers for a fraction of price, esp. on energy efficient hardware like Apple.