Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5 • Hacker News Board

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#161

Post by jtbaker » Sun, Sep 07, 2025, 7:37 PM UTC

anonym29 wrote: post #159

jtbaker wrote: post #140

Earlier quoted context omitted.

The 128 GB Strix Halo system was tempting me, but I think I'm going to hold out for the Medusa Point memory bandwidth gains to expand my cluster setup. I have a Mac Mini M4 Pro 64GB that does quite well with inference on the Qwen3 models, but is hell on networking with my home K3s cluster, which going deeper on is half the fun of this stuff for me.

>The 128 GB Strix Halo system was tempting me, but I think I'm going to hold out for the Medusa Point I was initially thinking this way too, but I realized a 128GB Strix Halo system would make an excellent addition to my homelab / LAN even once it's no longer the star of the stable for LLM inference - i.e. I will probably get a Medusa Halo system as well once they're available. My other devices are Zen 2 (3600x) / Ze…

That is sweet! The extent of my cluster is a few Pis that talk to the Mac Mini over the LAN for inference stuff, that I could definitely use some headroom on. I tried to integrate it into the cluster directly by running k3s in colima - but to join an existing cluster via IP, I had to run colima in host networking mode - so any pods on the mini that were trying to do CoreDNS networking were hitting collisions with mDNSResponder when dialing port 53 for DNS. Finally decided that the macs are nice machines but not a good fit for a member of a cluster.

Love that AMD seems to be closing the gap on the performance _and_ power efficiency of Apple Silicon with the latest Ryzen advancements. Seems like one of these new miniPCs would be a dream setup to run a bunch of data and AI centric hobby projects on - particularly workloads like geospatial imagery processing in addition to the LLM stuff. Its a fun time to be a tinkerer!

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#162

Post by rao-v » Mon, Sep 08, 2025, 7:16 AM UTC

jerrysievert wrote: post #130

rao-v wrote: post #123

Nice! Cheap RK3588 boards come with 15GB of LPDDR5 RAM these days and have significantly better performance than the Pi 5 (and often are cheaper). I get 8.2 tokens per second on a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB). I need to try two of them in parallel ... should be significantly faster than this even at Q6.

> a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB) fantastic! what are you using to run it, llama.cpp? I have a few extra opi5's sitting around that would love some extra usage

Yup! Build and ignore KleinAI and Vulkan etc. I’ve found that a clean CPU only build is optimal

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#163

Post by rao-v » Mon, Sep 08, 2025, 7:20 AM UTC

ThatPlayer wrote: post #150

rao-v wrote: post #123

Nice! Cheap RK3588 boards come with 15GB of LPDDR5 RAM these days and have significantly better performance than the Pi 5 (and often are cheaper). I get 8.2 tokens per second on a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB). I need to try two of them in parallel ... should be significantly faster than this even at Q6.

Is that using the NPU on that board? I know it's possible to use those too.

It is possibly (superb subreddit) but painful to convert a modern model and takes ages for them to be supported. The NPU is energy efficient but no faster than CPU for generation (and has lousy software support).

I’m mostly interested in the NPu to run a vision head in parallel with an LLM to speed up time to first token with VLLMs (kinda want to turn them into privacy safe vision devices for consumer use cases)

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#164

Post by rldjbpin » Mon, Sep 08, 2025, 8:27 AM UTC

how would llm-d [1] work compared to distributed-llama? is the overhead or configuration too much to work with for simple setups?

[1] https://github.com/llm-d/llm-d/

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#165

Post by ThatPlayer » Mon, Sep 08, 2025, 12:44 PM UTC

rao-v wrote: post #163

ThatPlayer wrote: post #150

Earlier quoted context omitted.

Is that using the NPU on that board? I know it's possible to use those too.

It is possibly (superb subreddit) but painful to convert a modern model and takes ages for them to be supported. The NPU is energy efficient but no faster than CPU for generation (and has lousy software support). I’m mostly interested in the NPu to run a vision head in parallel with an LLM to speed up time to first token with VLLMs (kinda want to turn them into privacy safe vision devices for consumer use cases)

Since my comment, I remembered I had a RK3588 board, a Rock 5B, and tried llama.cpp CPU over that, and performance was not amazing. But also I realized this is LPDDR4X, so don't get the cheapest RK3558 boards. My Orange Pi 5 is actually worse. This one has LPDDR4. Looking at the rest of Orange Pi's line-up, they don't actually have a board with both LPDDR5 and 32GB, only 16GB or LPDDR4(X).

Using llama-bench, and Llama 2 7B Q4_0 like https://github.com/ggml-org/llama.cpp/discussions/10879 how does yours compare? Cuz I'm also comparing it with a few a few Ryzen 5 3000 Series mini-pcs for less than 150$, and that gets 8 t/s on this list and I've gotten myself

With my Rock 5B and this bench, I get 3.65 t/s. On my Orange Pi 5 (not B) 8GB LPDDR4 (not X), I get 2.44 t/s.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#166

Post by supportengineer » Mon, Sep 08, 2025, 5:59 PM UTC

Aurornis wrote: post #99

supportengineer wrote: post #40

Earlier quoted context omitted.

[flagged]

Parent here. Kids have a lot of time and do a lot of different things. Some times it rains or snows or we’re home sick. Kids can (and will) do a lot of different things and it’s good to have options.

Kids can play in the rain and in the snow.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#167

Post by supportengineer » Mon, Sep 08, 2025, 6:00 PM UTC

ugh123 wrote: post #57

supportengineer wrote: post #40

Earlier quoted context omitted.

[flagged]

What about a kid who lives in an urban area without parks?

My snarky answer would be that a lack of parks presents a clear and present danger to the kid.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#168

Post by Bombthecat » Mon, Sep 08, 2025, 6:04 PM UTC

benreesman wrote: post #129

ivape wrote: post #21

Earlier quoted context omitted.

It just came to my attention that the 2021 M1 Max 64gb is less than $1500 used. That’s 64gb of unified memory at regular laptop prices, so I think people will be well equipped with AI laptops rather soon. Apple really is #2 and probably could be #1 in AI consumer hardware.

Ryzen AI 9 395+ with 64MB of LPDDR5 is 1500 new in a ton of factors and 2k with 128. If I have 1500 for a unified memory inference machine I'm probably not getting a Mac. It's not a bad choice per se, llama.cpp supports that harware extremely well, but a modern Ryzen APU at the same price is more of what I want for that use case, with the M1 Mac youre paying for a Retina display and a bunch of stuff unrelated to infe…

Ryzen 9 doesn't exist in Europe

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#169

Post by pseudosavant » Mon, Sep 08, 2025, 6:45 PM UTC

geerlingguy wrote: post #102

pseudosavant wrote: post #96

Earlier quoted context omitted.

The Raspberry Pi 2 Zero is as fast as a Pi 3, way smaller, and only costs $13 I think. The high end Pis aren’t $25 though.

The Pi 4 is still fine for a lot of low end use cases and starts at $35. The Pi 5 is in a harder position. I think the CM5 and Pi 500 are better showcases for it than the base model.

Between the microcontrollers, Zero models, the Pi 4, and the Pi 5, they have quite a full-range from very inexpensive and low power to moderate price/performance SBCs.

One of the bigger problems with Pi 5, is that many of the classic Pi use cases don't benefit from more CPU than the Pi 4 had. PCIe is nice, but you might as well go CM5 if you want something like that. The 16GB model would be more interesting if it had the GPU/bandwidth to do AI/tokens at a decent rate, but it doesn't.

I still think using any other brand of SBC is an exercise in futility though. Raspberry Pi products have the community, support, ecosystem behind them that no other SBC can match.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#170

Post by th0ma5 » Tue, Sep 09, 2025, 12:20 AM UTC

exe34 wrote: post #87

pdntspa wrote: post #63

Earlier quoted context omitted.

You can evaluate it as anecdote. How do I know you have the level of experience necessary to spot these kinds of problems as they arise? How do I know you're not just another AI booster with financial stake poisoning the discussion? We could go back and forth on this all day.

you got very defensive. it was a useful question - they were asking in terms of using a local LLM, so at best they might be in the business of selling raspberry pis, not proprietary LLMs.

Yeah to me it more poisonous that people reflexively believe any pushback must be wrong because people feel empowered regardless of any measurement that may point out that people only get (maybe) out of LLM models what they put into them, and even then we can't be sure. That this situation exists and people have been primed with a complete triangulation of all the arguments just simply isn't healthy and we should demand independent measurements instead of the fumbling in the dark of the current model measurements... Or admit that measuring them isn't helpful and like a parent maybe alluded to, can only be described as anecdote and there is no discernable difference between many models.