Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

41–50 of 276 posts

Re: Experimenting with Local LLMs on macOS

#41

Post by slacka »

coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

I too found that interesting that Apple's Neural Engine doesn't work with local LLMs. Seems like Apple, AMD, and Intel are missing the AI boat by not properly supporting their NPUs in llama.cpp. Any thoughts on why this is?

Re: Experimenting with Local LLMs on macOS

#42

Post by dragonwriter »

punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

Well, a lot of what is possible with local models depends on what your local hardware is, but docling is a pretty good example of a library that can use local models (VLMs instead of regular LLMs) “under the hood” for productive tasks.

Re: Experimenting with Local LLMs on macOS

#43

Post by HankStallone »

simonh wrote: post #18
daoboy wrote: post #12

I'm running Hermes Mistral and the very first thing it did was start hallucinating. I recently started an audio dream journal and want to keep it private. Set up whisper to transcribe the .wav file and dump it in an Obsidian folder. The plan was to put a local llm step in to clean up the punctuation and paragraphs. I entered instructions to clean the transcript without changing or adding anything else. Hermes respond…

It’s often been assumed that accuracy and ‘correctness’ would be easy to implement on computers because they operate on logic, in some sense. It’s originality and creativity that would be hard, or impossible because it’s not logical. Science Fiction has been full of such assumptions. Yet here we are, the actual problem is inventing new heavy enough training sticks to beat our AIs out of constantly making stuff up and…

The worst news I've seen about AI was a study that said the major ones get 40% of their references from Reddit (I don't know how they determined that). That explains the cloying way it tries to be friendly and supportive, too.

Re: Experimenting with Local LLMs on macOS

#44

Post by craftkiller »

Jemaclus wrote: post #32
craftkiller wrote: post #23

Earlier quoted context omitted.

It doesn't seem exactly like what they are describing. The end-user interface is what they are describing but it sounds like they want the actual LLM to run in the browser (perhaps via webgpu compute shaders). Open WebUI seems to rely on some external executor like ollama/llama.cpp, which naturally can still be self-hosted but they are not executing INSIDE the browser.

Does that even exist? It's basically what they described but with some additional installation? Once you install it, you can select the LLM on disk and run it? That's what they asked for. Maybe I'm misunderstanding something.

Apparently it does, though I'm learning about it for the first time in this thread also. Personally, I just run llama.cpp locally in docker-compose with anythingllm for the UI but I can see the appeal of having it all just run in the browser.

  https://github.com/mlc-ai/web-llm
  https://github.com/ngxson/wllama

Re: Experimenting with Local LLMs on macOS

#45

Post by JumpCrisscross »

bigyabai wrote: post #37
JumpCrisscross wrote: post #31

I don't think we're anywhere close to running cutting-edge LLMs on our phones or laptops. What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)

> $10 to 20k for a home LLM device isn't ridiculous. At that point you are almost paying more than the datacenter does for inference hardware.

> At that point you are almost paying more than the datacenter does for inference hardware

Of course. You and I don't have their economies of scale.

Re: Experimenting with Local LLMs on macOS

#46

Post by Jemaclus »

craftkiller wrote: post #44
Jemaclus wrote: post #32

Earlier quoted context omitted.

Does that even exist? It's basically what they described but with some additional installation? Once you install it, you can select the LLM on disk and run it? That's what they asked for. Maybe I'm misunderstanding something.

Apparently it does, though I'm learning about it for the first time in this thread also. Personally, I just run llama.cpp locally in docker-compose with anythingllm for the UI but I can see the appeal of having it all just run in the browser. https://github.com/mlc-ai/web-llm https://github.com/ngxson/wllama

Oh, interesting. Well, TIL.

Re: Experimenting with Local LLMs on macOS

#47

Post by brokencode »

JumpCrisscross wrote: post #31

I don't think we're anywhere close to running cutting-edge LLMs on our phones or laptops. What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)

Not sure about the Mac Pro, since you pay a lot for the big fancy case. The Studio seems more sensible.

And of course Nvidia and AMD are coming out with options for massive amounts of high bandwidth GPU memory in desktop form factors.

I like the idea of having basically a local LLM server that your laptop or other devices can connect to. Then your laptop doesn’t have to burn its battery on LLM work and it’s still local.

Re: Experimenting with Local LLMs on macOS

#48

Post by hu3 »

bigyabai wrote: post #36
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

Qwen3 A3B (in my experience) writes code as-good-as ChatGPT 4o and much better than GPT-OSS.

I just tested Qwen3 A3B vs ChatGPE a random prompt from my head and:

> Please write a C# middleware to block requests from browser agents that contain any word in a specified list of words: openai, grok, gemini, claude.

I used ChatpGPT 4o from GitHub Copilot inside VSCode. And Qwen3 A3B from here: https://deepinfra.com/Qwen/Qwen3-30B-A3B

ChatGPT 4o was considerably better. Less verbose and less unnecessary abstractions.

Re: Experimenting with Local LLMs on macOS

#49

Post by wslh »

coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

I find surprising that you can also do that from the browser (e.g. WebLLM). I imagine that in the near future we will run these engines locally for many use cases, instead of via APIs.

Re: Experimenting with Local LLMs on macOS

#50

Post by bigyabai »

hu3 wrote: post #48
bigyabai wrote: post #36

Earlier quoted context omitted.

Qwen3 A3B (in my experience) writes code as-good-as ChatGPT 4o and much better than GPT-OSS.

I just tested Qwen3 A3B vs ChatGPE a random prompt from my head and: > Please write a C# middleware to block requests from browser agents that contain any word in a specified list of words: openai, grok, gemini, claude. I used ChatpGPT 4o from GitHub Copilot inside VSCode. And Qwen3 A3B from here: https://deepinfra.com/Qwen/Qwen3-30B-A3B ChatGPT 4o was considerably better. Less verbose and less unnecessary abstractio…

You want the 2507 update of the model, I think the one you used is ~8-10 months out-of-date.
Post reply on HN