Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

github.com

Post reply on HN
Print view View original

121–130 of 171 posts

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#121

Post by ekianjo »

rs186 wrote: post #44
ryukoposting wrote: post #25

Earlier quoted context omitted.

I'd love to hook my development tools into a fully-local LLM. The question is context window and cost. If the context window isn't big enough, it won't be helpful for me. I'm not gonna drop $500 on RPis unless I know it'll be worth the money. I could try getting my employer to pay for it, but I'll probably have a much easier time convincing them to pay for Claude or whatever.

$500 gives you about 6 RPi 5 8GB or 4 16GB, excluding accessories or other necessary equipment to get this working. You'll be much better off spending that money on something else more useful.

Raspberry Pis going up in price make them very unattractive since there is a wealth of cheap second used better hardware out there such as NUCs with Celerons

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#122

Post by MalikTerm »

evilduck wrote: post #78
croes wrote: post #73

Earlier quoted context omitted.

What about AMD Ryzen AI Max+ 395 Mini PCs with upto 128GB unified memory?

Their memory bandwidth is the problem. 256 GB/s is really, really slow for LLMs. Seems like at the consumer hardware level you just have to pick your poison or what one factor you care about most. Macs with a Max or Ultra chip can have good memory bandwidth but low compute, but also ultra low power consumption. Discrete GPUs have great compute and bandwidth but low to middling VRAM, and high costs and power consumpti…

It's an underwhelming product in an annoying market segment, but 256GB/s really isn't that bad when you look at the competition. 150GB/s from hex channel DDR4, 200GB/s from quad channel DDR5, or around 256GB/s from Nvidia Digits or M Pro (that you can't get in the 128GB range). For context it's about what low-mid range GPUs provide, and 2.5-5x the bandwidth of the 50/100 GB/s memory that most people currently have.

If you're going with a Mac Studio Max you're going to be paying twice the price for twice the memory bandwidth, but the kicker is you'll be getting the same amount of compute as the AMD AI chips have which is going to be comparable to a low-mid range GPU. Even midrange GPUs like the RX 6800 or RTX 3060 are going to have 2x the compute. When the M1 chips first came out people were getting seriously bad prompt processing performance to the point that it was a legitimate consideration to make before purchase, and this was back when local models could barely manage 16k of context. If money wasn't a consideration and you decided to get the best possible Mac Studio Ultra, 800GB/s won't feel like a significant upgrade when it still takes 1 minute to process every 80k of uncached context that you'll absolutely be using on 1m context models.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#123

Post by rao-v »

Nice! Cheap RK3588 boards come with 15GB of LPDDR5 RAM these days and have significantly better performance than the Pi 5 (and often are cheaper).

I get 8.2 tokens per second on a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB). I need to try two of them in parallel ... should be significantly faster than this even at Q6.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#124

Post by mmastrac »

kristianp wrote: post #113
mmastrac wrote: post #31

Is the network the bottleneck here at all? That's impressive for a gigabit switch.

Does the switch use more power than the 4 pis?

Modern GB switches are pretty efficient (<10W for sure), I think a Pi might be 4-5W.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#125

Post by 9991 »

fragmede wrote: post #69
echelon wrote: post #3

This is really impressive. If we can get this down to a single Raspberry Pi, then we have crazy embedded toys and tools. Locally, at the edge, with no internet connection. Kids will be growing up with toys that talk to them and remember their stories. We're living in the sci-fi future. This was unthinkable ten years ago.

If a raspberry pi can do all that, imagine the toys Bill Gates' grandkids have access to! We're at the precipice of having a real "A Young Lady's Illustrated Primer" from The Diamond Age.

Bill Gates' grandkids will be playing with wooden blocks.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#126

Post by giancarlostoro »

dotancohen wrote: post #106
giancarlostoro wrote: post #67

Earlier quoted context omitted.

Sometimes you buy a pi for one project start on it buy another for a different project, before you know it none are complete and you have ten Raspberry Pis lying around across various generations. ;)

Arduino hobbist, same issue. Though I must admit to first noticing the trend decades before discovering Arduino when I looked at the stack of 289, 302, and 351W intake manifolds on my shelf and realised that I need the width of the 351W manifold but the fuel injection of the 302. Some things just never change.

I have different model Raspberry Pi's and I'm having a hard time justifying buying a 5... but if I can run LLMs off one or two... I just might. I guess what the next Raspberry Pi needs is a genuinely impressive GPU that COULD run small AI models, so people will start cracking at it.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#127

Post by vient »

trebligdivad wrote: post #93
dingdingdang wrote: post #4

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…

On my (single) AMD 3950x running entirely in CPU (llama -t32 -dev none), I was getting 14 tokens/s running Qwen3-Coder-30B-A3B-Instruct-IQ4_NL.gguf last night. Which is the best I've had out of a model that doesn't feel stupid.

For reference, I get 29 tokens/s with the same model using 12 threads on AMD 9950X3D. Guess it is 2x faster because AVX-512 is 2x faster on Zen 5, roughly speaking. Somewhat unexpectedly, increasing number of threads decreases performance, 16 threads already perform slightly worse and with 32 threads I only get 26.5 tokens/s.

On 5090 same model produces ~170 tokens/s.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#128

Post by benreesman »

rthnbgrredf wrote: post #8
dingdingdang wrote: post #4

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

If Moore's Law is Ending leaks are to be believed, there are going to be 24GB GDDR7 5080 Super and maybe even 5070 Super Ti variants in the 1k (MSRP) range and one assumes fast Blackwell NVFP4 Tensor Cores.

Depends on what you're doing, but at FP4 that goes pretty far.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#129

Post by benreesman »

ivape wrote: post #21
rthnbgrredf wrote: post #8

Earlier quoted context omitted.

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

It just came to my attention that the 2021 M1 Max 64gb is less than $1500 used. That’s 64gb of unified memory at regular laptop prices, so I think people will be well equipped with AI laptops rather soon. Apple really is #2 and probably could be #1 in AI consumer hardware.

Ryzen AI 9 395+ with 64MB of LPDDR5 is 1500 new in a ton of factors and 2k with 128. If I have 1500 for a unified memory inference machine I'm probably not getting a Mac. It's not a bad choice per se, llama.cpp supports that harware extremely well, but a modern Ryzen APU at the same price is more of what I want for that use case, with the M1 Mac youre paying for a Retina display and a bunch of stuff unrelated to inference.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#130

Post by jerrysievert »

rao-v wrote: post #123

Nice! Cheap RK3588 boards come with 15GB of LPDDR5 RAM these days and have significantly better performance than the Pi 5 (and often are cheaper). I get 8.2 tokens per second on a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB). I need to try two of them in parallel ... should be significantly faster than this even at Q6.

> a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB)

fantastic! what are you using to run it, llama.cpp? I have a few extra opi5's sitting around that would love some extra usage

Post reply on HN