[dead]
Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
91–100 of 149 posts
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#92Earlier quoted context omitted.
This is how progress happens, someone gets to 3t/s, the next person gets to6/s and eventually we get to 100t/s. People like this person are laying the foundations.
There is a limit how much you can squeeze out of given hardware. Betting on it being closer to 100t/s than 6t/s is only that, a bet.
On my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM.
The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here:
1. the organization of the data being sent to the GPU to optimize throughput;
2. the speed at which the GPU can read that data from its VRAM;
3. how many triangles it can process in parallel by using individual compute units.
I suspect that given parallel improvements for neural networks, we'll see similar improvements:
1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU;
2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication;
3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#93Earlier quoted context omitted.
So every script kiddie gets its Mythos to hack sides and scammers don’t need AI services anymore. That time won’t be as much fun as you think
By that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited. We'll be fine.
No, CVEs are on average still actively being exploited for far longer than 5 years. The oldest CVEs on CISA’s KEV (Known Exploited Vulnerabilities) catalog are from early to mid 2000s.
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#94I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
> wearing the hard drive Do reads wear out nvme? Iirc only writes do
only writes and honestly that’s a bit overblown they last a long time
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#95Earlier quoted context omitted.
Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.
You could do it but there would be no point, The only advantage over would be power consumption. And it would be quite expensive. At the rate models are improving, it would be obsolete in six months.
Yes, right now it would be obsolete in six months, but I also must add that this never stopped crypto miners from making new ASICs.
However, with how useful Kimi is right now - at some point if someone makes a dedicated hardware board with "good enough" model for daily tasks - that would be a very sought after commodity.
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#96The web and connecting to other services is very important for almost all of my use cases. While I believe we are going to get better and faster models, the web index is certainly not downloadable and maintainable for 99.99% of the folks who are able to use local models. Any good solutions exist?
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#97Thank you for using TurboFieldfare as a starting point for this project and thank you for mentioning it at the README. I am glad it inspired more people to explore area of on-device AI further!
[dead]
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#98Earlier quoted context omitted.
There is a limit how much you can squeeze out of given hardware. Betting on it being closer to 100t/s than 6t/s is only that, a bet.
On my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM. The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here: 1. the organization of the data being sent to the GPU to optimize throughput; 2. the speed at which the GPU can read that data from its VRAM; 3. how many triangles it can process in parallel by using individu…
If you've got X GB of weights in slow access memory (be it RAM vs VRAM or SSD vs RAM) and Y GB of fast memory then no matter what, if you want to use them you'll need to transfer X-Y GB and will be bound by memory throughput. You can try to reduce number of activated weights, but how much can be gained that way is speculative so far.
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#99I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk. In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of t…
> Most people are already used to rely on the internet on basically everything
FAANG has trained them well. "Normal" people also use chatGPT & co. to ask mundane questions that qwen27b could easily answer (random trivia, writing emails, correcting translations etc), why do these simple requests need to be processed on a B300 on server on the other side of the planet? There is no reason.
And this being hackernews, I would expect people here to go against the grain in this regard...maybe even promote local llms to "normies".
Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
#100"As far as we know, that is the first time a model of this class has run natively on a phone.". I feel like I've seen a similar statement on a lot of these streaming weight projects. 400b model on an iPhone: https://x.com/anemll/status/2035901335984611412
[dead]