Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

github.com

Post reply on HN
Print view View original

51–60 of 149 posts

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#51

Post by fragmede »

kaashif wrote: post #50
jack_pp wrote: post #38

Earlier quoted context omitted.

Isn't there such a thing as low hanging fruit? Aren't we already approaching theoretical physical limits? We're at 2nm

(1) Yes. (2) Are you saying that you think we're at the limits of computing in general, or that specific technology? We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily. But that is...not a low hanging fruit.

> We are nowhere near that.

Nowhere?

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#52

Post by kennywinker »

selcuka wrote: post #28
kennywinker wrote: post #7

Earlier quoted context omitted.

Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well

I'm pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.

Sure. One could. But then one wouldn’t be in control of every step of the process.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#53

Post by IsTom »

fsuts wrote: post #37
brrrrrm wrote: post #2

this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck. half an hour to process 10k tokens on an M5 seems... not great

This is how progress happens, someone gets to 3t/s, the next person gets to6/s and eventually we get to 100t/s. People like this person are laying the foundations.

There is a limit how much you can squeeze out of given hardware. Betting on it being closer to 100t/s than 6t/s is only that, a bet.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#54

Post by submeta »

gpt5 wrote: post #29
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk. In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of t…

I don’t understand the downvotes. It’s perfectly reasonable to disagree, but the argument is laid out clearly. Why not engage with it instead?

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#55

Post by xyzsparetimexyz »

anon373839 wrote: post #36
gpt5 wrote: post #29

Earlier quoted context omitted.

Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk. In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of t…

I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.

A lot of the compute that iPhones do either has high bandwidth requirements (photo editing etc) or low latency requirements (games etc). LLMs are an interesting case where unless you're feeding in images, it's just text

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#56

Post by cvak »

jack_pp wrote: post #38
apimade wrote: post #20

Earlier quoted context omitted.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Isn't there such a thing as low hanging fruit? Aren't we already approaching theoretical physical limits? We're at 2nm

we are not at 2nm, we just call it that.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#57

Post by zozbot234 »

petu wrote: post #25
wat10000 wrote: post #5

Earlier quoted context omitted.

Isn’t it only writes that kill drives?

There's read disturb on SSDs, enough reads will eventually force controller to rewrite the cell and it's neighbours. Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.

Read disturb ought to be quite rare, especially on a fresh drive that was written only once or a handful of times (WORM-like usage). Practically, it's not likely to be an issue even with very heavy read workloads.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#58

Post by sh3rl0ck »

arjie wrote: post #10
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.

We'd bought 4 x $11K Mac Studios at my college and via exo, we had Kimi K2.5 at 30 TPS.

Not too wild an idea!

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#59

Post by sallymander »

"As far as we know, that is the first time a model of this class has run natively on a phone.".

I feel like I've seen a similar statement on a lot of these streaming weight projects.

400b model on an iPhone: https://x.com/anemll/status/2035901335984611412

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#60

Post by apimade »

root_axis wrote: post #39
apimade wrote: post #20

Earlier quoted context omitted.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Ok, now do memory capacity and bandwidth - the things that actually constraint local LLMs.

8800 GTX in 2006: 768 MB of GDDR3, with 86.4 GB/s of theoretical memory bandwidth.

GTX 1080 in 2016: 8 GB of GDDR5X, with 320 GB/s.

RTX 5090 in 2026: 32 GB of GDDR7, with 1.792 TB/s.

This is fun, what's next?!

PCI 8.0 is breaking 1TB/s, GDDR7 is 1TB/s.

With just the _current_ timeline, things are looking like they'll compress once we get over this initial lump.

Post reply on HN