Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

171–180 of 276 posts

Re: Experimenting with Local LLMs on macOS

#171

Post by saagarjha »

giancarlostoro wrote: post #92
coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

One does not simply put a 5090 into an existing chip.

Re: Experimenting with Local LLMs on macOS

#172

Post by pmarreck »

alexashka wrote: post #164
aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

[flagged]

Oh look, it's a poor, green-text Google apologist who thinks phones with preinstalled crapware, an energy management model that doesn't stop any app from saturating your bandwidth, CPU or battery draw, and a security model that ensures you stand a good chance of becoming part of a crypto farm or botnet just because you downloaded an emulator from a third-party app store, means you have above an 80 IQ! LOL, way to virtue-signal your poverty, bro. These are tough times, I get it... But the first 2 Android phones I ever tried, I crashed within 5 minutes just by... get this... turning on their fucking Bluetooth. WHAT QUALITY. More like "what Chinese shovelware," amirite?

(How does it feel? Literally turning around your inane opinion back onto you.)

Re: Experimenting with Local LLMs on macOS

#173

Post by ghilston »

rukuu001 wrote: post #162
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

I'm running Gemma3-270M locally (MLX). I got a Python script that pulls down emails based on a whitelist and summarises them. The 270M model does a good job of this. This is running in a terminal. It means I barely look at my email during the day.

Any willingness to share this script? I've been working on some code to ingest things and summarize for them and I haven't gotten to email just yet.

Re: Experimenting with Local LLMs on macOS

#174

Post by saagarjha »

j45 wrote: post #129
jus3sixty wrote: post #126

An awful lot of Monday morning quarterback CEOs are here running their mouths about what Tim Cook should do or what they would do. Chill out with the extremely confident ignorance. Tim Cook brought Apple to a billion dollars in free cash he doesn’t need to ride the hype train. Also let’s not forget they are first and foremost designers of hardware and the arms race is only getting started.

Not sure I can think of anything that is more performant per watt for LLMs than Apple Silicon.

A datacenter GPU is going to be an order of magnitude more efficient.

Re: Experimenting with Local LLMs on macOS

#177

Post by moduspol »

utyop22 wrote: post #163
elAhmo wrote: post #144

Earlier quoted context omitted.

I think shareholders are fine with Tim Cook as a CEO.

I sometimes read posts on here and just laugh. Its easy to sit in the armchair and say "just be a visionary bro" when they forget Tim worked under Steve for awhile before his death - he has some sense and understanding of what it takes to get a great product out of the door. Nvidia is generating a lot of revenue, sure - but what is the downstream impact on its customers with the hardware? All they have right now is n…

This. I wouldn’t exactly give them bonus points for the handling of Apple Intelligence, but beyond that, they’ve taken a much more measured and evidence-based approach to LLMs than the rest of big tech.

If it ends up that we are in a bubble and it pops, Apple may be among the least impacted in big tech.

Re: Experimenting with Local LLMs on macOS

#178

Post by y1n0 »

elpakal wrote: post #166
woadwarrior01 wrote: post #101

Earlier quoted context omitted.

MLX does not support the ANE. https://github.com/ml-explore/mlx/issues/18

Yes it does. That’s just an issue with stale and incorrect information. Here are the docs https://opensource.apple.com/projects/mlx/

Nothing in that documentation says anything about the Apple Neural Engine. MLX runs on the GPU.

Re: Experimenting with Local LLMs on macOS

#179

Post by giancarlostoro »

alexashka wrote: post #164
aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

[flagged]

Doesn't Google sell $2000 phones? I really dont get the argument here.

Re: Experimenting with Local LLMs on macOS

#180

Post by logicprog »

zargon wrote: post #150
simonw wrote: post #60

Earlier quoted context omitted.

Right now you can run some of the best available open weight models on a 512GB Mac Studio, which retails for around $10,000. Here's Qwen3-Coder-480B-A35B-Instruct running at 24 tokens/second at 4bit: https://twitter.com/awnihannun/status/1947771502058672219 and Deep Seek V3 0324 in 4-bit at 20 toks/sec https://twitter.com/awnihannun/status/1904177084609827054 You can also string two 512GB Mac Studios together using M…

What these tweets about Apple silicon never show you: waiting 20+ minutes for it to ingest 32k context tokens. (Probably a lot longer for these big models.)

Yeah, I bought a used Mac Studio (an M1, to be fair, but still a Max and things haven't changed since) hoping to be able to run a decent LLM on it, and was sorely disappointed thanks to the prompt processing speed especially.
Post reply on HN