Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

191–200 of 276 posts

Re: Experimenting with Local LLMs on macOS

#191

Post by spease »

elAhmo wrote: post #144
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I think shareholders are fine with Tim Cook as a CEO.

Yes. And everyone is glossing over the benefit of unified memory for LLM applications. Apple may not have the models, but it has customer goodwill, a platform, and the logistical infrastructure to roll them out. It probably even has the cash to buy some AI companies outright; maybe not the big ones (for a reasonable amount, anyway) but small to midsize ones with domain-specific models that could be combined.

Not to mention the “default browser” leverage it has with with iPhones, iPods, and watches.

Re: Experimenting with Local LLMs on macOS

#192

Post by jychang »

seanmcdirmid wrote: post #190
flutas wrote: post #187

Earlier quoted context omitted.

Or someone told Tim "we can save $XYZ per phone if we switch to custom designed silicon, and potentially expand it to Mac as well so we no longer have Intel overheating our Macbooks." He was after-all more of an operations guy than a product guy before moving into the CEO role.

The unified GPU and unified memory design was pretty important. They just didn’t go and replace intel, they replaced AMD/NVIDIA also. The GPUs in high end Apple silicon are even good enough for mid model inference, and unified memory makes it somewhat cost effective…that advantage probably wasn’t planned and comes from just a lot of good execution and smart R&D.

To be fair, Apple HATES Nvidia after the 8400M and 9400M debacle. They probably saw replacing Nvidia as a bigger benefit than replacing Intel.

Re: Experimenting with Local LLMs on macOS

#193

Post by jychang »

elpakal wrote: post #166
woadwarrior01 wrote: post #101

Earlier quoted context omitted.

MLX does not support the ANE. https://github.com/ml-explore/mlx/issues/18

Yes it does. That’s just an issue with stale and incorrect information. Here are the docs https://opensource.apple.com/projects/mlx/

None of that uses the ANE.

Re: Experimenting with Local LLMs on macOS

#194

Post by balder1991 »

jftuga wrote: post #71

I have a macbook air M4 with 32 GB. What LM Studio models would you recommend for: * General Q&A * Specific to programming - mostly Python and Go. I forgot the command now, but I did run a command that allowed MacOS to allocate and use maybe 28 GB of RAM to the GPU for use with LLMs.

You’ll certainly find better answers on /r/LocalLlama in Reddit for this.

Re: Experimenting with Local LLMs on macOS

#196

Post by mg »

simonw wrote: post #114
mg wrote: post #76

Earlier quoted context omitted.

This might be a misunderstanding. Did you see the "button that the user can click to select a model from their file system" part of my comment? I tried some of the demos of transformers.js but they all seem to load the model from a server. Which is super slow. I would like to have a page the lets me use any model I have on my disk.

Oh sorry, I missed that bit. I got Codex + GPT-5 to modify that Llama chat example to implement the "load from local directory" pattern. It appears to work. First you'll need to grab the checkout of the local model (~1.3GB): git lfs install git clone https://huggingface.co/onnx-community/Llama-3.2-1B-Instruct-q4f16 Then visit this page: https://static.simonwillison.net/static/2025/llama-3.2-webgp... - in Chrome or Fi…

Awesome!

Bookmarked. I will surely try it out once FireFox or Chromium on Linux support WebGPU in their default config.

Re: Experimenting with Local LLMs on macOS

#197

Post by kristopolous »

jeffybefffy519 wrote: post #145
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

Gemma3 is pretty useful on a long haul flight without internet

kimi v2 by moonshot. Give it a go

Re: Experimenting with Local LLMs on macOS

#198

Post by oblio »

segmondy wrote: post #102
oblio wrote: post #39

Earlier quoted context omitted.

I think the point was that for example for programming, people perceive state of the art LLMs as being net positive contributors, at least for mainstream programming languages and tasks, and I guess local LLMs aren't net positive contributors (i.e. an experienced programmer can build the same thing at least as fast when using an LLM).

I know this is false, DeepSeekv3.1, GLM4.5, KimiK2-0905, Qwen-235B are all solid open models. Last night, I vibed rough 1300 lines of C server code in about an hour. 0 compilation error, ran without errors and got the job done. I want to meet this experienced programmer that can knock out 1300 lines of C code in an hour.

Can you run any of those models without $20 000 worth of hardware that uses as much power and makes as much noise as a small factory?

Re: Experimenting with Local LLMs on macOS

#199

Post by vorticalbox »

punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

I keep a lot of notes, all my thoughts feelings both happy and sad, things I’ve done, etc. in obsidian. These are deeply personal and I don’t want this going to a cloud provider even if they “say” they don’t train on my chats.

I forget a lot of things so I feed these into chromeDB and then use a LLM to chat with all my notes.

I’ve started using abliterated models which have their refusal removed [0]

Other use case is for work. I work with financial data and I have created an mcp that automates some of my job. Running model locally allows me to not worry about the information I feed it.

[0] https://github.com/Sumandora/remove-refusals-with-transforme...

Post reply on HN