Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

111–120 of 276 posts

Re: Experimenting with Local LLMs on macOS

#111

Post by svachalek »

slacka wrote: post #41
coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

I too found that interesting that Apple's Neural Engine doesn't work with local LLMs. Seems like Apple, AMD, and Intel are missing the AI boat by not properly supporting their NPUs in llama.cpp. Any thoughts on why this is?

I think I saw something that got Ollama to run models on it? But it only works with tiny models. Seems like the neural engine is extremely power efficient but not fast enough to do LLMs with billions of parameters.

Re: Experimenting with Local LLMs on macOS

#112

Post by aurareturn »

bigyabai wrote: post #52
slacka wrote: post #41

Earlier quoted context omitted.

I too found that interesting that Apple's Neural Engine doesn't work with local LLMs. Seems like Apple, AMD, and Intel are missing the AI boat by not properly supporting their NPUs in llama.cpp. Any thoughts on why this is?

Most NPUs are almost universally too weak to use for serious LLM inference. Most of the time you get better performance-per-watt out of GPU compute shaders, the majority of NPUs are dark silicon. Keep in mind - Nvidia has no NPU hardware because that functionality is baked-into their GPU architecture. AMD, Apple and Intel are all in this awkward NPU boat because they wanted to avoid competition with Nvidia and contin…

Apple is in this NPU boat because they are optimized for mobile first.

Nvidia does not optimize for mobile first.

AMD and Intel were forced by Microsoft to add NPUs in order to sell “AI PCs”. Turns out the kind of AI that people want to run locally can’t run on an NPU. It’s too weak like you said.

AMD and Intel both have matmul acceleration directly in their GPUs. Only Apple does not.

Re: Experimenting with Local LLMs on macOS

#113

Post by jasonjmcghee »

I think the best models around right now that most people can fit some quantization on their computer if it's a apple silicon Mac or gaming PC would be:

For non-coding: Qwen3-30B-A3B-Instruct-2507 (or the thinking variant, depending on use case)

For coding: Qwen3-Coder-30B-A3B-Instruct

---

If you have a bit more vram, GLM-4.5-Air or the full GLM-4.5

Re: Experimenting with Local LLMs on macOS

#114

Post by simonw »

mg wrote: post #76
simonw wrote: post #64

Earlier quoted context omitted.

Transformers.js ( https://huggingface.co/docs/transformers.js/en/index ) is this. Some demos (should work in Chrome and Firefox on Windows, or Firefox Nightly on macOS and Linux): https://huggingface.co/spaces/webml-community/llama-3.2-webg... loads a 1.24GB Llama 3.2 q4f16 ONNX build https://huggingface.co/spaces/webml-community/janus-pro-webg... loads a 2.24 GB DeepSeek Janus Pro model which is multi-modal for outp…

This might be a misunderstanding. Did you see the "button that the user can click to select a model from their file system" part of my comment? I tried some of the demos of transformers.js but they all seem to load the model from a server. Which is super slow. I would like to have a page the lets me use any model I have on my disk.

Oh sorry, I missed that bit.

I got Codex + GPT-5 to modify that Llama chat example to implement the "load from local directory" pattern. It appears to work.

First you'll need to grab the checkout of the local model (~1.3GB):

  git lfs install
  git clone https://huggingface.co/onnx-community/Llama-3.2-1B-Instruct-q4f16
Then visit this page: https://static.simonwillison.net/static/2025/llama-3.2-webgp... - in Chrome or Firefox Nightly.

Now click "Browse folder" and select the folder you just checked out with Git.

Click the confusing "Upload" confirmation (it doesn't upload anything, just opens those files in the current browser session).

Now click "Load local model" - and you should get a full working chat interface.

Code is here: https://github.com/simonw/transformers.js-examples/commit/cd...

Here's the full Codex session that I used to build this: https://gist.github.com/simonw/3c46c9e609f6ee77367a760b5ca01...

I ran Codex against the https://github.com/huggingface/transformers.js-examples/tree... folder and prompted:

> Modify this application such that it offers the user a file browse button for selecting their own local copy of the model file instead of loading it over the network. Provide a "download model" option too.

Then later:

> Build the production app and then make it available on localhost somehow

And:

> Uncaught (in promise) Error: Invalid configuration detected: both local and remote models are disabled. Fix by setting `env.allowLocalModels` or `env.allowRemoteModels` to `true`.

And:

> Add a bash script which will build the application such that I can upload a folder called llama-3.2-webgpu to http://static.simonwillison.net/static/2025/llama-3.2-webgpu... and http://static.simonwillison.net/static/2025/llama-3.2-webgpu... will serve the app

(Note that this doesn't allow you to use any model on your machine, but it proves that it's possible.)

Re: Experimenting with Local LLMs on macOS

#115

Post by JumpCrisscross »

dghlsakjg wrote: post #93
JumpCrisscross wrote: post #53

Earlier quoted context omitted.

> Not sure about the Mac Pro, since you pay a lot for the big fancy case. The Studio seems more sensible Oh wow, a maxed out Studio could run a 600B parameter model entirely in memory. Not bad for $12k. There may be a business in creating the software that links that box to an app on your phone.

That software is an HTTP request, no? Any number of AI apps allow you to specify a custom endpoint. As long as your AI server accepts connections to the internet, you're gravy.

> That software is an HTTP request, no?

You and I could write it. Most folks couldn’t. If AI plateaus, this would be a good hill to have occupied.

Re: Experimenting with Local LLMs on macOS

#116

Post by all2 »

jasonjmcghee wrote: post #113

I think the best models around right now that most people can fit some quantization on their computer if it's a apple silicon Mac or gaming PC would be: For non-coding: Qwen3-30B-A3B-Instruct-2507 (or the thinking variant, depending on use case) For coding: Qwen3-Coder-30B-A3B-Instruct --- If you have a bit more vram, GLM-4.5-Air or the full GLM-4.5

Note that Qwen3 and Deepseek are hobbled in Ollama; they cannot use tools as the tool portion of the system prompt is missing.

Recommendation: use something else to run the model. Ollama is convenient, but insufficient for tool use for these models.

Re: Experimenting with Local LLMs on macOS

#118

Post by wer232essf »

coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

[flagged]

Re: Experimenting with Local LLMs on macOS

#120

Post by nomel »

segmondy wrote: post #102
oblio wrote: post #39

Earlier quoted context omitted.

I think the point was that for example for programming, people perceive state of the art LLMs as being net positive contributors, at least for mainstream programming languages and tasks, and I guess local LLMs aren't net positive contributors (i.e. an experienced programmer can build the same thing at least as fast when using an LLM).

I know this is false, DeepSeekv3.1, GLM4.5, KimiK2-0905, Qwen-235B are all solid open models. Last night, I vibed rough 1300 lines of C server code in about an hour. 0 compilation error, ran without errors and got the job done. I want to meet this experienced programmer that can knock out 1300 lines of C code in an hour.

Without knowing what you were doing with that 1300 lines of code, there's not much insight that can be had from this.
Post reply on HN