Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

251–260 of 276 posts

Re: Experimenting with Local LLMs on macOS

#251

Post by Damogran6 »

frontsideair wrote: post #204
Damogran6 wrote: post #94

Oddly, my 2013 MacPro (Trashcan) runs LLMs pretty well, mostly because 64Gb of old school RAM is, like, $25.

I'm interested in this, my impression was that the newer chips have unified memory and high memory bandwidth. Do you do inference on the CPU or the external GPU?

I don't, I'm a REALLY light user. smaller LLMs work pretty well. I used a 40gb LLM and it was _pokey_, but it worked, and switching them is pretty easy. This is a 12 core Xeon with 64Gb RAM...my M4 mini is....okay with smaller LLMs, I have a Ryzen 9 with a RTX3070ti that's the best of the bunch, but none of this holds a candle to people that spend real money to experiment in this field.

Re: Experimenting with Local LLMs on macOS

#252

Post by billbrown »

moduspol wrote: post #177
utyop22 wrote: post #163

Earlier quoted context omitted.

I sometimes read posts on here and just laugh. Its easy to sit in the armchair and say "just be a visionary bro" when they forget Tim worked under Steve for awhile before his death - he has some sense and understanding of what it takes to get a great product out of the door. Nvidia is generating a lot of revenue, sure - but what is the downstream impact on its customers with the hardware? All they have right now is n…

This. I wouldn’t exactly give them bonus points for the handling of Apple Intelligence, but beyond that, they’ve taken a much more measured and evidence-based approach to LLMs than the rest of big tech. If it ends up that we are in a bubble and it pops, Apple may be among the least impacted in big tech.

Toyota did this with the EV mania until they lost their nerve and got rid of Toyoda as CEO. I hope Apple doesn't fall into the same trap. (I never thought Toyota would give in either.)

Re: Experimenting with Local LLMs on macOS

#253

Post by otterley »

alt227 wrote: post #239
aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

Apple silicon does not compete well in multicore spaces. People seem to think that because it can run single core things really well on a laptop, it can do anything. Servers regularly have 100-200 cpu cores maxing out of rapid fire threads. This is not what Apple silicon excels at. On top of that, it only performs so well on consumer devices because they control the hardware and OS and can tune both together. Creatin…

> Apple silicon does not compete well in multicore spaces.

Can you elaborate on this? Maybe with some useful metrics?

Re: Experimenting with Local LLMs on macOS

#255

Post by swiftcoder »

alt227 wrote: post #239
aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

Apple silicon does not compete well in multicore spaces. People seem to think that because it can run single core things really well on a laptop, it can do anything. Servers regularly have 100-200 cpu cores maxing out of rapid fire threads. This is not what Apple silicon excels at. On top of that, it only performs so well on consumer devices because they control the hardware and OS and can tune both together. Creatin…

> This is not what Apple silicon excels at

Not at the moment, no. I feel like the Apple silicon team probably would rise to that challenge though

Re: Experimenting with Local LLMs on macOS

#256

Post by cma »

Someone wrote: post #70
slacka wrote: post #41

Earlier quoted context omitted.

I too found that interesting that Apple's Neural Engine doesn't work with local LLMs. Seems like Apple, AMD, and Intel are missing the AI boat by not properly supporting their NPUs in llama.cpp. Any thoughts on why this is?

I guess that hardware doesn’t make things faster (¿yet?). If so I guess they would have mentioned it in https://machinelearning.apple.com/research/core-ml-on-device... . That is updated for Sequoia and says “This technical post details how to optimize and deploy an LLM to Apple silicon, achieving the performance required for real time use cases. In this example we use Llama-3.1-8B-Instruct, a popular mid-size LLM, an…

If it uses a lot less power it could still be a win for some use cases, like while on battery you might still want to run transformer based speech to text, RTX voice-like microphone denoising, image generation/infill in photo editing programs. In some use cases like RTX-voice like stuff during multiplayer gaming, you might want the GPU free to run the game even if it still suffers some memory bandwidth impact from having it running.

Re: Experimenting with Local LLMs on macOS

#257

Post by reddit_clone »

svachalek wrote: post #111
slacka wrote: post #41

Earlier quoted context omitted.

I too found that interesting that Apple's Neural Engine doesn't work with local LLMs. Seems like Apple, AMD, and Intel are missing the AI boat by not properly supporting their NPUs in llama.cpp. Any thoughts on why this is?

I think I saw something that got Ollama to run models on it? But it only works with tiny models. Seems like the neural engine is extremely power efficient but not fast enough to do LLMs with billions of parameters.

I am running Ollama with 'SimonPu/Qwen3-Coder:30B-Instruct_Q4_K_XL' on a M4 pro MBP with 48 GB of memory.

From Emacs/gptel, it seems pretty fast.

I have never used the proper hosted LLMS, so I don't have a direct comparison. But the above LLM answered coding questions in a handful of seconds.

The cost of memory (and disk) upgrades in apple machines is exorbitant.

Re: Experimenting with Local LLMs on macOS

#258

Post by ActorNightly »

punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

Smaller models require a lot more direction, a.k.a system prompt engineering, and sometimes custom wrappers . For example Gemma models are very eager to generate code even if you tell them not to.

Re: Experimenting with Local LLMs on macOS

#259

Post by bigyabai »

pmarreck wrote: post #243
bigyabai wrote: post #181

Earlier quoted context omitted.

It feels like you are particularly insecure and didn't need to spout that any more than the parent did.

Nope. I just want to show a douche what it looks like.

Oh, good. Your comment was materially indistinct from someone who took the "iPad and Vision Pro are toys" thing a bit too personally.

Re: Experimenting with Local LLMs on macOS

#260

Post by zargon »

alt227 wrote: post #240
logicprog wrote: post #180

Earlier quoted context omitted.

Yeah, I bought a used Mac Studio (an M1, to be fair, but still a Max and things haven't changed since) hoping to be able to run a decent LLM on it, and was sorely disappointed thanks to the prompt processing speed especially.

No offense to you personally, but I find it very funny when people hear marketing copy for a product and think it can do anything they said it can. Apple silicon is still just a single consumer grade chip. It might be able to run certain end user software well, but it cannot replace a server rack of GPUs.

I don’t think this is a fair take in this particular situation. My comment is in response to Simon Willison, who has a very popular blog in the LLM space. This isn’t company marketing copy; it’s trusted third parties spreading this misleading information.
Post reply on HN