Earlier quoted context omitted.
> I'd love to hook my development tools into a fully-local LLM. Karpathy said in his recent talk, on the topic of AI developer-assistants: don't bother with less capable models. So ... using an rpi is probably not what you want.
Mind linking to "his recent talk"? There's a lot of videos of him so it's a bit difficult to find what's most recent.
Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
111–120 of 171 posts
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#112Everything runs on a π if you quantize it enough! I'm curious about the applications though. Do people randomly buy 4xRPi5s that they can now dedicate to running LLMs?
For $500 you may as well spend an extra $100 and get a Mac mini with an m4 chip and 256gb of ram and avoid the headaches of coordinating 4 machines.
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#113Is the network the bottleneck here at all? That's impressive for a gigabit switch.
Does the switch use more power than the 4 pis?
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#114Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#115Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#116Earlier quoted context omitted.
Their memory bandwidth is the problem. 256 GB/s is really, really slow for LLMs. Seems like at the consumer hardware level you just have to pick your poison or what one factor you care about most. Macs with a Max or Ultra chip can have good memory bandwidth but low compute, but also ultra low power consumption. Discrete GPUs have great compute and bandwidth but low to middling VRAM, and high costs and power consumpti…
But for matrix multiplication, isn't compute more important, as there are N³ multiplications but just N² numbers in a matrix? Also I don't think power consumption is important for AI. Typically you do AI at home or in the office where there is lot of electricity.
>But for matrix multiplication, isn't compute more important, as there are N³ multiplications but just N² numbers in a matrix?
Being able to quickly calculate a dumb or unreliable result because you're VRAM starved is not very useful for most scenarios. To run capable models you need VRAM, so high VRAM and lower compute is usually more useful than the inverse (a lot of both is even better, but you need a lot of money and power for that).
Even in this post with four RPis, the Qwen3 30 A3B is still an MOE model and not a dense model. It runs fast with only 3B active parameters and can be parallelized across computers but it's much less capable than a dense 30B model running on a single GPU.
> Also I don't think power consumption is important for AI. Typically you do AI at home or in the office where there is lot of electricity.
Depends on what scale you're discussing. If you want to get similar VRAM as a 512GB Mac Studio Ultra with a bunch of Nvidia GPUs like RTX 3090 cards you're not going to be able to run that on a typical American 15 AMP circuits, you'll trip a breaker half way there.
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#117Earlier quoted context omitted.
On my (single) AMD 3950x running entirely in CPU (llama -t32 -dev none), I was getting 14 tokens/s running Qwen3-Coder-30B-A3B-Instruct-IQ4_NL.gguf last night. Which is the best I've had out of a model that doesn't feel stupid.
How much RAM it is using by the way? I see 30B, but without knowing precision it is unclear how much memory one needs.
Q4 is usually around 4.5 bits per parameter but can be more as some layers are quantised to a higher precision, which would suggest 30 billion * 4.5 bit = 15.7GB, but the quant the GP is using is 17.3GB and 19.7GB for the article. Add around 20-50% overhead for various things and then some % for each 1k of tokens in the context and you're probably looking at no more than 32GB. If you're using something like llama.cpp which can offload some of the model to the GPU you'll still get decent performance even on a 16gb VRAM GPU.
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#118Earlier quoted context omitted.
I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.
What about AMD Ryzen AI Max+ 395 Mini PCs with upto 128GB unified memory?
Works very well and very fast with this Qwen3 30B A3B model.
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#119Earlier quoted context omitted.
How much RAM it is using by the way? I see 30B, but without knowing precision it is unclear how much memory one needs.
Q4 is usually around 4.5 bits per parameter but can be more as some layers are quantised to a higher precision, which would suggest 30 billion * 4.5 bit = 15.7GB, but the quant the GP is using is 17.3GB and 19.7GB for the article. Add around 20-50% overhead for various things and then some % for each 1k of tokens in the context and you're probably looking at no more than 32GB. If you're using something like llama.cpp…
Sounds close! top says my llama is using 17.7G virt, 16.6G resident with:
./build/bin/llama-cli -m /discs/fast/ai/Qwen3-Coder-30B-A3B-Instruct-IQ4_NL.gguf --jinja -ngl 99 --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --presence-penalty 1.0 -t 32 -dev none
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#120Earlier quoted context omitted.
> I'd love to hook my development tools into a fully-local LLM. Karpathy said in his recent talk, on the topic of AI developer-assistants: don't bother with less capable models. So ... using an rpi is probably not what you want.
> Karpathy said in his recent talk, on the topic of AI developer-assistants: don't bother with less capable models. Interesting because he also said the future is small "cognitive core" models: > a few billion param model that maximally sacrifices encyclopedic knowledge for capability. It lives always-on and by default on every computer as the kernel of LLM personal computing. https://xcancel.com/karpathy/status/1938…
It's not at all trivial to build a "small but highly capable" model. Sacrificing world knowledge is something that can be done, but only to an extent, and that isn't a silver bullet.
For an LLM, size is a virtue - the larger a model is, the more intelligent it is, all other things equal - and even aggressive distillation only gets you this far.
Maybe with significantly better post-training, a lot of distillation from a very large and very capable model, and extremely high quality synthetic data, you could fit GPT-5 Pro tier of reasoning and tool use, with severe cuts to world knowledge, into a 40B model. But not into a 4B one. And it would need some very specific training to know when to fall back to web search or knowledge databases, or delegate to a larger cloud-hosted model.
And if we had the kind of training mastery required to pull that off? I'm a bit afraid of what kind of AI we would be able to train as a frontier run.