Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
1–10 of 171 posts
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#2distributed-llama is great, I just wish it would work with more models. I've been happy with ease of setup and its ongoing maintenance compared to Exo, and performance vs llama.cpp RPC mode.
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#3This is really impressive.
If we can get this down to a single Raspberry Pi, then we have crazy embedded toys and tools. Locally, at the edge, with no internet connection.
Kids will be growing up with toys that talk to them and remember their stories.
We're living in the sci-fi future. This was unthinkable ten years ago.
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#4Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with limited budgets but large PC fleets ;)
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#5[flagged]
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#6distributed-llama is great, I just wish it would work with more models. I've been happy with ease of setup and its ongoing maintenance compared to Exo, and performance vs llama.cpp RPC mode.
Any pointers to what is SOTA for cluster of hosts with CUDA GPUs but not enough vram for full weights, yet 10Gbit low latency interconnects?
If that problem gets solved, even if for only a batch approach that enables parallel batch inference resulting in high total token/s but low per session, and for bigger models, then it would he a serious game changer for large scale low cost AI automation without billions capex. My intuition says it should be possible, so perhaps someone has done it or started on it already.
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#7This is really impressive. If we can get this down to a single Raspberry Pi, then we have crazy embedded toys and tools. Locally, at the edge, with no internet connection. Kids will be growing up with toys that talk to them and remember their stories. We're living in the sci-fi future. This was unthinkable ten years ago.
[flagged]
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#8Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…
I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering.
Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#9This is really impressive. If we can get this down to a single Raspberry Pi, then we have crazy embedded toys and tools. Locally, at the edge, with no internet connection. Kids will be growing up with toys that talk to them and remember their stories. We're living in the sci-fi future. This was unthinkable ten years ago.
[flagged]
this is very pessimistic take
there are lot of bad people on internet too, does that make internet is a mistake ???
Noo, the people are not the tool
Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
#10[flagged]
13/s is not slow. Q4 is not bad. The models that run on phones are never 30B or anywhere close to that.