Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

github.com

Post reply on HN
Print view View original

31–40 of 149 posts

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#31

Post by harrouet »

Mistletoe wrote: post #9
AHASIC wrote: post #8

I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.

What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.

I could definitely image Apple embedding a kind of LLM-optimized FPGA: slow to load (update) an LLM, but blazing fast at computing tokens.

Who needs memory when your model is set in silicon ?

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#32

Post by xbmcuser »

apimade wrote: post #20
arjie wrote: post #10

Earlier quoted context omitted.

Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#33

Post by formerly_proven »

apimade wrote: post #20
arjie wrote: post #10

Earlier quoted context omitted.

Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Except the 499$ of a 1080 GTX inflation-adjusted only buys you a 5070 or 5070 Ti even by MSRP.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#35

Post by zmmmmm »

apimade wrote: post #20
arjie wrote: post #10

Earlier quoted context omitted.

Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Sadly while the FLOPS are increasing nicely, total graphics memory is stalled in consumer cards by comparison.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#36

Post by anon373839 »

gpt5 wrote: post #29
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk. In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of t…

I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#37

Post by fsuts »

brrrrrm wrote: post #2

this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck. half an hour to process 10k tokens on an M5 seems... not great

This is how progress happens, someone gets to 3t/s, the next person gets to6/s and eventually we get to 100t/s.

People like this person are laying the foundations.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#38

Post by jack_pp »

apimade wrote: post #20
arjie wrote: post #10

Earlier quoted context omitted.

Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Isn't there such a thing as low hanging fruit?

Aren't we already approaching theoretical physical limits? We're at 2nm

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#39

Post by root_axis »

apimade wrote: post #20
arjie wrote: post #10

Earlier quoted context omitted.

Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Ok, now do memory capacity and bandwidth - the things that actually constraint local LLMs.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#40

Post by hdgvhicv »

selcuka wrote: post #28
kennywinker wrote: post #7

Earlier quoted context omitted.

Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well

I'm pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.

Domestic electricity is free nowadays, certainly for most of the year, as solar plus battery covers your usage for a tiny percentage of the cost of your house.
Post reply on HN