Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

github.com

Post reply on HN
Print view View original

71–80 of 149 posts

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#71

Post by mihaaly »

gpt5 wrote: post #29
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk. In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of t…

This is a nice personal point of view. And you don't care much about privacy or being exposed to the conduct of others completely, do you?

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#72

Post by kaelwd »

foxrider wrote: post #65
apimade wrote: post #20

Earlier quoted context omitted.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.

Only 8B currently but it's been done: https://taalas.com/products/

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#73

Post by croes »

dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

So every script kiddie gets its Mythos to hack sides and scammers don’t need AI services anymore.

That time won’t be as much fun as you think

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#74

Post by naasking »

HPsquared wrote: post #70
SJC_Hacker wrote: post #68

Earlier quoted context omitted.

You could do it but there would be no point, The only advantage over would be power consumption. And it would be quite expensive. At the rate models are improving, it would be obsolete in six months.

Power consumption and latency are very important on mobile

They're important everywhere of course, but especially on mobile. If AI researchers figure out how to offload knowledge and expertise from reasoning weights, then a core reasoning ASIC linked to the knowledge would totally rock.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#75

Post by dtj1123 »

flaburgan wrote: post #48
apimade wrote: post #20

Earlier quoted context omitted.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Yeah but here you describing the opposite phenomenon. You're saying that the hardware is going to become cheaper and more powerful with the years, to the point a current State of the Art model from today will run on a normal consumer hardware in ten years. What people are trying to do now is the opposite, optimize the software as much as possible so that it does not need the best hardware but the normal one we curren…

No, both of these things can happen in parallel.

The suggestion is that a 1T model could be made to run on cheap consumer hardware of the future.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#76

Post by hobofan »

gpt5 wrote: post #29
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk. In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of t…

> Those two combined means that unless LLMs reach the state of 'good enough'

Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.

Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#77

Post by cromka »

croes wrote: post #73
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

So every script kiddie gets its Mythos to hack sides and scammers don’t need AI services anymore. That time won’t be as much fun as you think

By that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited.

We'll be fine.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#78

Post by apimade »

kaelwd wrote: post #72
foxrider wrote: post #65

Earlier quoted context omitted.

Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.

Only 8B currently but it's been done: https://taalas.com/products/

The only _public_ example we know.

This is definitely being done with private models by HFT/quant firms, data processing agencies/orgs (large intelligence agencies, _every_ data analytics org, etc).

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#79

Post by myrmidon »

dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets.

Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.

I expect similar trends with AI; I'd expect to get decent, human comparable capability with I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#80

Post by pizza234 »

dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

> but this is how progress happens.

This is progress in the same way that a man climbing a tree is making progress toward reaching the moon.

This project is essentially the MoE-of-the day, with some platform-related optimizations.

> Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

That won't happen. Projects like this just give the illusion that that will be possible.

Post reply on HN