Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

231–240 of 276 posts

Re: Experimenting with Local LLMs on macOS

#231

Post by brookst »

giancarlostoro wrote: post #92
coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

Under Cook, Apple’s market cap has increased 10x, at a CAGR of 18%.

Do you really think that they need something different? As a shareholder would you bet on your vision of focusing on server parts?

Re: Experimenting with Local LLMs on macOS

#232

Post by brookst »

jbverschoor wrote: post #124
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

Local llm.. everybody is scared of privacy.. many people don’t want to buy subscriptions (still). Just sell a proper HomePod with 64GB-128GB ram, which handles everything including your personal LLM, Time Machine if needed, back to Mac (Tailscale/zerotier) + they can compete efficiently with the other. Cloud providers.

It’s a mistake to generalize from the HN population.

Most people don’t care about privacy (see: success of Facebook and TikTok). Most people don’t care about subscriptions (see: cable TV, Netflix).

There may be a niche market for a local inference device that costs $1000 and has to be replaced every year or two during the early days of AI, but it’s not a market with decent ROI for Apple.

Re: Experimenting with Local LLMs on macOS

#233

Post by giancarlostoro »

jbs789 wrote: post #97
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

Sounds like you’ve got a solid handle on things - go do it!

Give me a majority share in AAPL if that's what you want ;)

Re: Experimenting with Local LLMs on macOS

#234

Post by giancarlostoro »

saagarjha wrote: post #171
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

One does not simply put a 5090 into an existing chip.

Not what I am suggesting. However, having trained a few different things on a modest M4 Pro chip (so not even their absolute most powerful chips mind you), and using it for local-first AI inference, I can see the value. A single server could serve an LLM for a small business and cost a lot less than running the same inference through a 5090 in terms of power usage.

I could also see universities giving this type of compute access to students for cheaper to work on more basic less resource intensive models.

Re: Experimenting with Local LLMs on macOS

#235

Post by DrAwdeOccarim »

theshrike79 wrote: post #226
DrAwdeOccarim wrote: post #160

Earlier quoted context omitted.

I adore Qwen 3 30b a3b 2507. Pretty easy to write an MCP to let us search the web with Brave API key. I run it on my Macbook Pro M3 Pro 36 GB.

What are you running it on that lets you connect tools to it?

LM Studio. I just vibe code the nodeJS code.

Re: Experimenting with Local LLMs on macOS

#236

Post by brookst »

segmondy wrote: post #102
oblio wrote: post #39

Earlier quoted context omitted.

I think the point was that for example for programming, people perceive state of the art LLMs as being net positive contributors, at least for mainstream programming languages and tasks, and I guess local LLMs aren't net positive contributors (i.e. an experienced programmer can build the same thing at least as fast when using an LLM).

I know this is false, DeepSeekv3.1, GLM4.5, KimiK2-0905, Qwen-235B are all solid open models. Last night, I vibed rough 1300 lines of C server code in about an hour. 0 compilation error, ran without errors and got the job done. I want to meet this experienced programmer that can knock out 1300 lines of C code in an hour.

I’m a mediocre C programmer on my best day and I assure you a highly competent programmer could probably use 200 lines of code to do what I achieve in 1300.

Just counting lines is not a good proxy for how much effort it would take a good programmer.

(And I am 100% pro LLM coding, just saying this isn’t a great argument)

Re: Experimenting with Local LLMs on macOS

#237

Post by brookst »

bigyabai wrote: post #51
JumpCrisscross wrote: post #45

Earlier quoted context omitted.

> At that point you are almost paying more than the datacenter does for inference hardware Of course. You and I don't have their economies of scale.

Then please excuse me for calling your one-man $10,000 inference device ridiculous.

How is it not impressive to be able to do something at quantity 1 for roughly the same price megacorps get at quantity 100,000?

Try building a F1 car at home. I guarantee your unit cost will be several orders of magnitude higher than the companies who make several a year.

Re: Experimenting with Local LLMs on macOS

#238

Post by ChrisMarshallNY »

moduspol wrote: post #177
utyop22 wrote: post #163

Earlier quoted context omitted.

I sometimes read posts on here and just laugh. Its easy to sit in the armchair and say "just be a visionary bro" when they forget Tim worked under Steve for awhile before his death - he has some sense and understanding of what it takes to get a great product out of the door. Nvidia is generating a lot of revenue, sure - but what is the downstream impact on its customers with the hardware? All they have right now is n…

This. I wouldn’t exactly give them bonus points for the handling of Apple Intelligence, but beyond that, they’ve taken a much more measured and evidence-based approach to LLMs than the rest of big tech. If it ends up that we are in a bubble and it pops, Apple may be among the least impacted in big tech.

Friend of mine, used to work for Apple.

He told me that a popular Apple saying is "We're late to the party, but always best-dressed."

I understand this. I'm not sure their choice of outfit has always been the best, but they have had enough success to continue making money.

Re: Experimenting with Local LLMs on macOS

#239

Post by alt227 »

aurareturn wrote: post #107
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

Apple silicon does not compete well in multicore spaces. People seem to think that because it can run single core things really well on a laptop, it can do anything. Servers regularly have 100-200 cpu cores maxing out of rapid fire threads. This is not what Apple silicon excels at.

On top of that, it only performs so well on consumer devices because they control the hardware and OS and can tune both together. Creating server hardware would mean allowing linux to be installed on it, and would need to run equally well. Apple would never put the development time into linux kernel/drivers to make this happen.

Re: Experimenting with Local LLMs on macOS

#240

Post by alt227 »

logicprog wrote: post #180
zargon wrote: post #150

Earlier quoted context omitted.

What these tweets about Apple silicon never show you: waiting 20+ minutes for it to ingest 32k context tokens. (Probably a lot longer for these big models.)

Yeah, I bought a used Mac Studio (an M1, to be fair, but still a Max and things haven't changed since) hoping to be able to run a decent LLM on it, and was sorely disappointed thanks to the prompt processing speed especially.

No offense to you personally, but I find it very funny when people hear marketing copy for a product and think it can do anything they said it can.

Apple silicon is still just a single consumer grade chip. It might be able to run certain end user software well, but it cannot replace a server rack of GPUs.

Post reply on HN