Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models on an M4 with 24GB memory

jola.dev

Post reply on HN
Print view View original

111–120 of 189 posts

Re: Running local models on an M4 with 24GB memory

#111

Post by 2ndorderthought »

guessmyname wrote: post #26
nu11ptr wrote: post #17

Still trying to understand if a Macbook Pro M5 Max with 128GB is likely going to be able to run coding models well enough that I can cancel my Codex, or even go down to the $20/month plan.

A 128GiB MacBook Pro in Canada is what, north of CAD $11k after tax? That’s around USD $7k. At $20/month for a cloud AI subscription, you’re looking at almost 30 years of service for the same money. How long do people realistically expect a laptop to stay competitive with SOTA local models? Especially in a space where model sizes, context windows, and inference requirements keep moving every year. And even if the har…

You can buy a used GPU for under 400 dollars if you already have a desktop and run qwen 3.6 a3b and for a majority of frontier tasks get by just fine. Why do you need to spend 10k on a laptop, we are swimming in ewaste.

Re: Running local models on an M4 with 24GB memory

#112

Post by 2ndorderthought »

reillyse wrote: post #36

so, interested how many people are running higher end AI models locally? Figure if I'm spending $800/month on tokens I can build a pretty beefy local machine for the cost of a few months spend - what is people's experience with say a $5k server custom built (and only for) running an AI model.

Check in with /r/localllama. There's 100gb vram set ups from complete ewaste to single 8gb GPU inference machines.depends on what you want and can afford

Re: Running local models on an M4 with 24GB memory

#113

Post by nu11ptr »

gabagool wrote: post #59
tom_ wrote: post #48

Earlier quoted context omitted.

But what if you were going to buy a laptop anyway? Obviously you can't do anything with less than 64 GBytes these days, so the question is just whether you go for the jump to 128. In the UK, it's currently an extra £800 to get a 128 GB vs the 64 GB equivalent. So that's more like 3 years of Claude - I think? - assuming current prices stay the same. Or: you might just feel like £800 isn't an unjustifiable amount of mo…

> Obviously you can't do anything with less than 64 GBytes these days I don't think that's true. Plenty of people can run basic workflows at 8GB on the MacBook Neo and most others are fine at 16 GB.

I am a developer, as many of us on here are. I currently have 32GB of RAM and am constantly fighting swap. 64GB would be min even w/o local model.

Re: Running local models on an M4 with 24GB memory

#117

Post by K0balt »

thot_experiment wrote: post #71
fg137 wrote: post #69

Earlier quoted context omitted.

Sure. Sample size = 1.

It may surprise you but over thousands of hours I have actually gathered more than one sample. EDIT: Here's another sample for ya. I went to the store to buy mixers and while I was out Gemma 4 31b got pretty far along with reverse engineering the bluetooth protocol of a desk thermometer I have. I forgot to turn on the web search tool, so it just went at it, writing more and more specific diagnostic logging/probing to…

What is your opinion on qwen 35b MOEvs qwen 27b dense?

Re: Running local models on an M4 with 24GB memory

#118

Post by dale_glass »

guessmyname wrote: post #26
nu11ptr wrote: post #17

Still trying to understand if a Macbook Pro M5 Max with 128GB is likely going to be able to run coding models well enough that I can cancel my Codex, or even go down to the $20/month plan.

A 128GiB MacBook Pro in Canada is what, north of CAD $11k after tax? That’s around USD $7k. At $20/month for a cloud AI subscription, you’re looking at almost 30 years of service for the same money. How long do people realistically expect a laptop to stay competitive with SOTA local models? Especially in a space where model sizes, context windows, and inference requirements keep moving every year. And even if the har…

Buy a Framework Desktop with 128 GB instead. It's half the price, and though I bought it for even less before RAM prices went crazy.

Re: Running local models on an M4 with 24GB memory

#120

Post by Galanwe »

busfahrer wrote: post #116

I am considering a M5 Pro (18/20C) Macbook with 64GB of RAM, but I'm having a really hard time finding benchmarks of real world models: Could somebody please provide some tokens-per-second numbers for example for Qwen 3.6 35B/A3B, specifically for Q4 and Q6 quants?

My advice: don't just look at tokens per second, but also at time to first token (TTFT).

The local inference space is leaning to MoE models, and a lot of them have decent tokens / second, but horrible TTFT.

Post reply on HN