Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

181–190 of 276 posts

Re: Experimenting with Local LLMs on macOS

#181

Post by bigyabai »

pmarreck wrote: post #172
alexashka wrote: post #164

Earlier quoted context omitted.

[flagged]

Oh look, it's a poor, green-text Google apologist who thinks phones with preinstalled crapware, an energy management model that doesn't stop any app from saturating your bandwidth, CPU or battery draw, and a security model that ensures you stand a good chance of becoming part of a crypto farm or botnet just because you downloaded an emulator from a third-party app store, means you have above an 80 IQ! LOL, way to vir…

It feels like you are particularly insecure and didn't need to spout that any more than the parent did.

Re: Experimenting with Local LLMs on macOS

#182

Post by zozbot234 »

coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

From reverse engineered information (in the context of Asahi Linux, which can have raw hardware access to the ANE) it seems that the M1/M2 Apple Neural Engine provides exclusively for statically scheduled MADD's of INT8 or FP16 values.[0] This wastes a lot of memory bandwidth on padding in the context of newer local models which generally are more heavily quantized.

(That is, when in-memory model values must be padded to FP16/INT8 this slashes your effective use of memory bandwidth, which is what determines token generation speed. GPU compute doesn't have that issue; one can simply de-quantize/pad the input in fast local registers to feed the matrix compute units, so memory bandwidth is used efficiently.)

The NPU/ANE is still potentially useful for lowering power use in the context of prompt pre-processing, which is limited by raw compute as opposed to the memory bandwidth bound of token generation. (Lower power usage in this context will save on battery and may help performance by avoiding power/thermal throttling, especially on passively-cooled laptops. So this is definitely worth going for.)

[0] Some historical information about bare-metal use of the ANE is available from the Whisper.cpp pull req: https://github.com/ggml-org/whisper.cpp/pull/1021 Even older information at: https://github.com/eiln/ane/tree/33a61249d773f8f50c02ab0b9fe... .

More extensive information at https://github.com/tinygrad/tinygrad/tree/master/extra/accel... (from the Tinygrad folks) seems to basically confirm the above.

(The jury is still out for M3/M4 which currently have no Asahi support - thus, no current prospects for driving the ANE bare-metal. Note however that the M3/Pro/Max ANE reported performance numbers are quite close to the M2 version, so there may not be a real improvement there either. M3 Ultra and especially the M4 series may be a different story.)

Re: Experimenting with Local LLMs on macOS

#183

Post by itsn0tm3 »

dghlsakjg wrote: post #149
JumpCrisscross wrote: post #115

Earlier quoted context omitted.

> That software is an HTTP request, no? You and I could write it. Most folks couldn’t. If AI plateaus, this would be a good hill to have occupied.

My point is, what is there to build? The person that is willing to buy that appliance is likely heavily overlapped with the person that is more than capable of pointing one of the dozens of existing apps at a custom domain. Everyone else will continue to just use app based subscriptions. Streaming platforms have plateaued (at best), but self hosted media appliances are still vanishingly rare. Why would AI buck the tr…

You don’t tell your media player company secrets ;)

I think there is a market here, solely based on actual data privacy. Not sure how big it is but I can see quite some companies have use for it.

Re: Experimenting with Local LLMs on macOS

#184

Post by dghlsakjg »

itsn0tm3 wrote: post #183
dghlsakjg wrote: post #149

Earlier quoted context omitted.

My point is, what is there to build? The person that is willing to buy that appliance is likely heavily overlapped with the person that is more than capable of pointing one of the dozens of existing apps at a custom domain. Everyone else will continue to just use app based subscriptions. Streaming platforms have plateaued (at best), but self hosted media appliances are still vanishingly rare. Why would AI buck the tr…

You don’t tell your media player company secrets ;) I think there is a market here, solely based on actual data privacy. Not sure how big it is but I can see quite some companies have use for it.

> You don’t tell your media player company secrets ;)

No, but my email provider has a de-facto repository of incredibly sensitive documents. When you put convenience and cost up against privacy, the market has proven over and over that no one gives a shit.

Re: Experimenting with Local LLMs on macOS

#185

Post by daemonologist »

coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

ONNX Runtime purports to support CoreML: https://onnxruntime.ai/docs/execution-providers/CoreML-Execu... , which gives a decent amount of compatibility for inference. I have no idea to what extent workloads actually end up on the ANE though.

(Unfortunately ONNX doesn't support Vulkan, which limits it on other platforms. It's always something...)

Re: Experimenting with Local LLMs on macOS

#186

Post by rukuu001 »

ghilston wrote: post #173
rukuu001 wrote: post #162

Earlier quoted context omitted.

I'm running Gemma3-270M locally (MLX). I got a Python script that pulls down emails based on a whitelist and summarises them. The 270M model does a good job of this. This is running in a terminal. It means I barely look at my email during the day.

Any willingness to share this script? I've been working on some code to ingest things and summarize for them and I haven't gotten to email just yet.

Watch this space. It’s pretty scrappy code and needs a cleanup. It also does other random stuff relating to calendar entries that I want to be reminded to appropriately prepare for.

But yes I’ll share, and I guess post an update in this thread?

Re: Experimenting with Local LLMs on macOS

#187

Post by flutas »

seanmcdirmid wrote: post #146
aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

I was pleasantly surprised Apple Silicon came out at all. Someone has their eye on long term vision at Apple at least, they just didn't do this on a whim.

Or someone told Tim "we can save $XYZ per phone if we switch to custom designed silicon, and potentially expand it to Mac as well so we no longer have Intel overheating our Macbooks."

He was after-all more of an operations guy than a product guy before moving into the CEO role.

Re: Experimenting with Local LLMs on macOS

#188

Post by redundantly »

bigyabai wrote: post #135
jbverschoor wrote: post #124

Earlier quoted context omitted.

Local llm.. everybody is scared of privacy.. many people don’t want to buy subscriptions (still). Just sell a proper HomePod with 64GB-128GB ram, which handles everything including your personal LLM, Time Machine if needed, back to Mac (Tailscale/zerotier) + they can compete efficiently with the other. Cloud providers.

> Just sell a proper HomePod with 64GB-128GB ram The same Homepod that almost sold as poorly as Vision Pro despite a $349.99 MSRP? Apple charges $400 to upgrade an M4 to 64GB and a whopping $1,200 for the 128GB upgrade. The consumer demand for a $800+ device like this is probably zilch, I can't imagine it's worth Apple's time to gussy up a nice UX or support it long-term. What you are describing is a Mac with extra s…

> The same Homepod that almost sold as poorly as Vision Pro despite a $349.99 MSRP?

The HomePod did poorly because competitor offerings with similar and better performing features were priced under $100. The difference in sound quality was not worth the >3x markup.

Re: Experimenting with Local LLMs on macOS

#190

Post by seanmcdirmid »

flutas wrote: post #187
seanmcdirmid wrote: post #146

Earlier quoted context omitted.

I was pleasantly surprised Apple Silicon came out at all. Someone has their eye on long term vision at Apple at least, they just didn't do this on a whim.

Or someone told Tim "we can save $XYZ per phone if we switch to custom designed silicon, and potentially expand it to Mac as well so we no longer have Intel overheating our Macbooks." He was after-all more of an operations guy than a product guy before moving into the CEO role.

The unified GPU and unified memory design was pretty important. They just didn’t go and replace intel, they replaced AMD/NVIDIA also. The GPUs in high end Apple silicon are even good enough for mid model inference, and unified memory makes it somewhat cost effective…that advantage probably wasn’t planned and comes from just a lot of good execution and smart R&D.
Post reply on HN