Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

221–230 of 276 posts

Re: Experimenting with Local LLMs on macOS

#221

Post by VagabundoP »

jbverschoor wrote: post #124
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

Local llm.. everybody is scared of privacy.. many people don’t want to buy subscriptions (still). Just sell a proper HomePod with 64GB-128GB ram, which handles everything including your personal LLM, Time Machine if needed, back to Mac (Tailscale/zerotier) + they can compete efficiently with the other. Cloud providers.

Have a team pushing out opitmised open source models. Over time this thing could become the house AI. Basically Star Treks computer.

Re: Experimenting with Local LLMs on macOS

#222

Post by theshrike79 »

dxetech wrote: post #34
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

There are situations where internet access is limited, or where there are frequent outages. An outdated LLM might be more useful than none at all. For example: my internet is out due to a severe storm, what safety precautions do I need to take?

Just use Kiwix: https://kiwix.org/en/

Re: Experimenting with Local LLMs on macOS

#223

Post by woooooo »

giancarlostoro wrote: post #92
coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

Not to mention, build a car with all that cash they have. Xiaomi makes awesome cars, Apple branded electric could scoop all the brand equity that Elon passed away.

Re: Experimenting with Local LLMs on macOS

#224

Post by theshrike79 »

brokencode wrote: post #47
JumpCrisscross wrote: post #31

I don't think we're anywhere close to running cutting-edge LLMs on our phones or laptops. What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)

Not sure about the Mac Pro, since you pay a lot for the big fancy case. The Studio seems more sensible. And of course Nvidia and AMD are coming out with options for massive amounts of high bandwidth GPU memory in desktop form factors. I like the idea of having basically a local LLM server that your laptop or other devices can connect to. Then your laptop doesn’t have to burn its battery on LLM work and it’s still loc…

It's really easy to whip up a simple box that runs local LLM for a whole home.

Marketing it though? Not doable.

Apple is pretty much the only company I see attempting this with some kind of AppleTV Pro.

Re: Experimenting with Local LLMs on macOS

#225

Post by theshrike79 »

all2 wrote: post #116
jasonjmcghee wrote: post #113

I think the best models around right now that most people can fit some quantization on their computer if it's a apple silicon Mac or gaming PC would be: For non-coding: Qwen3-30B-A3B-Instruct-2507 (or the thinking variant, depending on use case) For coding: Qwen3-Coder-30B-A3B-Instruct --- If you have a bit more vram, GLM-4.5-Air or the full GLM-4.5

Note that Qwen3 and Deepseek are hobbled in Ollama; they cannot use tools as the tool portion of the system prompt is missing. Recommendation: use something else to run the model. Ollama is convenient, but insufficient for tool use for these models.

Could you give a recommendation that works instead of saying what doesn't work?

Re: Experimenting with Local LLMs on macOS

#226

Post by theshrike79 »

DrAwdeOccarim wrote: post #160
jftuga wrote: post #71

I have a macbook air M4 with 32 GB. What LM Studio models would you recommend for: * General Q&A * Specific to programming - mostly Python and Go. I forgot the command now, but I did run a command that allowed MacOS to allocate and use maybe 28 GB of RAM to the GPU for use with LLMs.

I adore Qwen 3 30b a3b 2507. Pretty easy to write an MCP to let us search the web with Brave API key. I run it on my Macbook Pro M3 Pro 36 GB.

What are you running it on that lets you connect tools to it?

Re: Experimenting with Local LLMs on macOS

#227

Post by renmillar »

Miraste wrote: post #142
mrexroad wrote: post #123

Earlier quoted context omitted.

They did have Xserve back in the day. As great as Apple silicon is for running local llms along with being a general-purpose computing device, it’s not clear that Apple silicon have enough of a differentiating advantage over a rack of nvidia gpus to make it worthwhile in enterprise…

Strange to be saying this about Apple products, but its advantage is that it's way, way cheaper.

Would probably be different if NVIDIA viewed it as competition for data center market share

Re: Experimenting with Local LLMs on macOS

#228

Post by ghilston »

rukuu001 wrote: post #186
ghilston wrote: post #173

Earlier quoted context omitted.

Any willingness to share this script? I've been working on some code to ingest things and summarize for them and I haven't gotten to email just yet.

Watch this space. It’s pretty scrappy code and needs a cleanup. It also does other random stuff relating to calendar entries that I want to be reminded to appropriately prepare for. But yes I’ll share, and I guess post an update in this thread?

Okay will do. Yeah, I just finished my calendar reading code. I haven't prepared the data for my LLM ingestion yet. Sounds great, I'll refresh this thread in a few days

Re: Experimenting with Local LLMs on macOS

#229

Post by brookst »

aurareturn wrote: post #107
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

Is expansion to all possible markets really a sign of product vision? Windows is in everything from ATMs to servers to cheap laptops, and I am not sure it’s a better product for it OR that Microsoft makes more money that way. Certainly the support burden for a huge number of long tail applications is huge.

And I suppose we’re giving credit to other people for Watch, AirPods, Vision Pro?

Re: Experimenting with Local LLMs on macOS

#230

Post by simonw »

theshrike79 wrote: post #225
all2 wrote: post #116

Earlier quoted context omitted.

Note that Qwen3 and Deepseek are hobbled in Ollama; they cannot use tools as the tool portion of the system prompt is missing. Recommendation: use something else to run the model. Ollama is convenient, but insufficient for tool use for these models.

Could you give a recommendation that works instead of saying what doesn't work?

Try LM Studio or llama-server: https://simonwillison.net/2025/Aug/19/gpt-oss-with-llama-cpp...
Post reply on HN