Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

github.com

Post reply on HN
Print view View original

121–130 of 149 posts

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#121

Post by xprnio »

harrouet wrote: post #106
KeplerBoy wrote: post #66

Earlier quoted context omitted.

You don't an FPGA if you're taping out your own chips. But that is just a MMA accelerator with decent memory bandwidth. No secret sauce here.

I am talking about reconfigurable gates to implement an LLM in silicon, i.e. an FPGA...

What might the “parameters/layers to gates” ratio look like? My naive and uninformed guess would be that 1B+ parameter would also need a 1B+ gate FPGA, but according to google they typically range from tens of thousands to several million (which would still be a fraction of a billion).

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#123

Post by bluerooibos »

pizza234 wrote: post #80
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

> but this is how progress happens. This is progress in the same way that a man climbing a tree is making progress toward reaching the moon. This project is essentially the MoE-of-the day, with some platform-related optimizations. > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips. That won't happen. Projects like this just give the illus…

> That won't happen

I don't understand how anyone with any degree of technological or scientific literacy or knowledge of advances in the past 100-200 years can say something won't happen in the future.

I am absolutely certain it will - the only uncertainty is how long, but given the rate of progress, I'm guessing it won't be far away.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#124

Post by red_green_yell »

dghlsakjg wrote: post #115
dd8601fn wrote: post #41

Earlier quoted context omitted.

Especially if you’re Apple and your expertise and reputation is in making privacy conscious devices and cloud services that are exceptionally user friendly. Then you go to the AI labs and say, “We see you’ve spent a trillion dollars on that model… we’ll write you a check today to let us run it how we want.” Skips the ugly horse race and massive risk while focusing on what they do better than anyone. And they’re the o…

Apple has a real knack for not being first mover but being “best” mover. They didn’t invent tablets or mp3 players or thin laptops or ARM laptops, but once they entered those markets they absolutely became the definitional product for the entire category. We don’t talk about AVP.

Don’t forget AirPods, Apple TV, and Apple Watch. They were not first but they are currently best. The Alexa thing and AVP are notable fails.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#125

Post by piyh »

brrrrrm wrote: post #2

this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck. half an hour to process 10k tokens on an M5 seems... not great

SSD NAND reads are nearly infinite. Still makes me uncomfortable, but writing is what kills. There's a reason SSDs are rated by TBW, not TBR.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#126

Post by dgently7 »

harrouet wrote: post #31
Mistletoe wrote: post #9

Earlier quoted context omitted.

What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.

I could definitely image Apple embedding a kind of LLM-optimized FPGA: slow to load (update) an LLM, but blazing fast at computing tokens. Who needs memory when your model is set in silicon ?

why would apple make a chip that could be updated to improve the model when they could just sell you a better chip in the next years device?

on device llm gives apple the new "better camera" "better screen" race they need to keep people coming back for the latest.

for average users everything else is tapped out... screens, cameras wifi... all the core stuff is good enough now its hard to feel/see the difference model year to model year. embedded llm would let them ship something new and the on device ecosystem advantage is huge. especially as the gpt and claudes get ads and enshittified... the apple on device even if its less "capable" would be so compelling.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#127

Post by marci »

josu wrote: post #62
marci wrote: post #49

Earlier quoted context omitted.

Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...

I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?

Nothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks.

https://machinelearning.apple.com/research/introducing-third...

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#129

Post by tredre3 »

sbsbdbdbfndj wrote: post #82
throwawayffffas wrote: post #67

> One expectation to set honestly Hello Claude!

Let me just make sure first instead of guessing

That line was authored by Claude. There you go: https://github.com/leonickson1/Swiftlet/commit/3ac64020eadcb...
Post reply on HN