Unrelated but I really enjoyed the wavy text effect on “opinions” in the first paragraph
Thank you, it was the integral part of the whole post!
Experimenting with Local LLMs on macOS
201–210 of 276 posts
Re: Experimenting with Local LLMs on macOS
#202Every blog post or article about running local LLMs should include something about which hardware was used.
Good point, let me add a quick note.
Re: Experimenting with Local LLMs on macOS
#203Earlier quoted context omitted.
Apple is in this NPU boat because they are optimized for mobile first. Nvidia does not optimize for mobile first. AMD and Intel were forced by Microsoft to add NPUs in order to sell “AI PCs”. Turns out the kind of AI that people want to run locally can’t run on an NPU. It’s too weak like you said. AMD and Intel both have matmul acceleration directly in their GPUs. Only Apple does not.
Nvidia's approach works just fine on mobile. Devices like the Switch have complex GPGPU pipelines and don't compromise whatsoever on power efficiency. Nonetheless, Apple's architecture on mobile doesn't have to define how they approach laptops, destops and datacenters. If the mobile-first approach is limiting their addressable market, then maybe Tim's obsessing over the wrong audience?
MacBooks benefit from mobile optimization. Apple just needs to add matmul hardware acceleration into their GPUs.
Re: Experimenting with Local LLMs on macOS
#204Oddly, my 2013 MacPro (Trashcan) runs LLMs pretty well, mostly because 64Gb of old school RAM is, like, $25.
I'm interested in this, my impression was that the newer chips have unified memory and high memory bandwidth. Do you do inference on the CPU or the external GPU?
Re: Experimenting with Local LLMs on macOS
#205I have a macbook air M4 with 32 GB. What LM Studio models would you recommend for: * General Q&A * Specific to programming - mostly Python and Go. I forgot the command now, but I did run a command that allowed MacOS to allocate and use maybe 28 GB of RAM to the GPU for use with LLMs.
This is the command probably:
sudo sysctl iogpu.wired_limit_mb=184320
Re: Experimenting with Local LLMs on macOS
#206ollama is another good choice for this purpose. it's essentially a wrapper around llamacpp that adds easy downloading and management of running instances. it's great! also works on linux!
Ollama adding a paid cloud version made me postpone this post for a few weeks at least. I don't object them to make money, but it was hard to recommend a tool for local usage and make the first instruction to go to settings and enable airplane mode.
Luckily llama.cpp has come a long way and was at a point that I could easily recommend as the open source option instead.
Re: Experimenting with Local LLMs on macOS
#207The really though spot is finding a good model for your use case. I’ve a 16Gb MB and have been paralyzed by the many options. I’ve settle for a quantisied 14B Qwen for now, but no idea if this is a good idea.
14B Qwen was a good choice, but it became outdated a bit and seems like the new version of 4B surpassed it in benchmarks somehow.
It's a balancing game, how slow a token generation speed can you tolerate? Would you rather get an answer quick, or wait for a few seconds (or sometimes minutes) for reasoning?
For quick answers, Gemma 3 12B is still good. GPT-OSS 20B is pretty quick when reasoning is set to low, which usually doesn't think longer than one sentence. I haven't gotten much use out of Qwen3 4B Thinking (2507) but at least it's fast while reasoning.
Re: Experimenting with Local LLMs on macOS
#208Re: Experimenting with Local LLMs on macOS
#209Earlier quoted context omitted.
MLX does not support the ANE. https://github.com/ml-explore/mlx/issues/18
Yes it does. That’s just an issue with stale and incorrect information. Here are the docs https://opensource.apple.com/projects/mlx/
No, it categorically doesn't. Not just that, it's CPU support is quite lacking (fp32 only). Currently, there are two ways to support the ANE: CoreML and MPSGraph.
Re: Experimenting with Local LLMs on macOS
#210>I also use them for brain-dumping. I find it hard to keep a journal, because I find it boring, but when you’re pretending to be writing to someone, it’s easier. If you have friends, that’s much better, but some topics are too personal and a friend may not be available at 4 AM. I mostly ignore its responses, because it’s for me to unload, not to listen to a machine spew slop. I suggest you do the same, because we’re anthropomorphization machines and I’d rather not experience AI psychosis. It’s better if you don’t give it a chance to convince you it’s real. I could use a system prompt so it doesn’t follow up with dumb questions (or “YoU’Re AbSoLuTeLy CoRrEcT”s), but I never bothered as I already don’t read it.
Reads like someone starting to get their daily drinks, already using them for "company" and fun, and saying "I'm not an alcoholic, I can quit anytime".