Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
1–10 of 149 posts
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#2this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.
half an hour to process 10k tokens on an M5 seems... not great
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#3At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#4At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.
It says very prominently in the post: 4.5-5t/s for 80b on an M5
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#5At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.
Isn’t it only writes that kill drives?
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#6I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#7this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck. half an hour to process 10k tokens on an M5 seems... not great
Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#8I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#9I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#10I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.