Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

github.com

Post reply on HN
Print view View original

101–110 of 171 posts

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#101

Post by kasey_junk »

herval wrote: post #76
mycall wrote: post #61

Earlier quoted context omitted.

Two main factors are holding back the "killer app" for AI. Fix hallucinations and make agents more deterministic. Once these are in place, people will love AI when it can make them money somehow.

How does one “fix hallucinations” on an LLM? Isn’t hallucinating pretty much all it does?

Coding agents have shown how. You filter the output against something that can tell the llm when it’s hallucinating.

The hard part is identifying those filter functions outside of the code domain.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#102

Post by geerlingguy »

pseudosavant wrote: post #96
throaway920181 wrote: post #72

Earlier quoted context omitted.

It's sad that Pis are now so overpriced. They used to be fun little tinker boards that were semi-cheap.

The Raspberry Pi 2 Zero is as fast as a Pi 3, way smaller, and only costs $13 I think. The high end Pis aren’t $25 though.

The Pi 4 is still fine for a lot of low end use cases and starts at $35. The Pi 5 is in a harder position. I think the CM5 and Pi 500 are better showcases for it than the base model.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#103

Post by dotancohen »

ivape wrote: post #43
jeroenhd wrote: post #29

Earlier quoted context omitted.

Apple is leagues ahead of Microsoft with the whole AI PC thing and so far it has yet to mean anything. I don't think consumers care at all about running AI, let alone running AI locally. I'd try the whole AI thing on my work Macbook but Apple's built-in AI stuff isn't available in my language, so perhaps that's also why I haven't heard anybody mention it.

People don’t know what they want yet, you have to show it to them. Getting the hardware out is part of it, but you are right, we’re missing the killer apps at the moment. The very need for privacy with AI will make personal hardware important no matter what.

  > People don’t know what they want yet, you have to show it to them
Henry Ford famously quipped that had he asked his customers what they wanted, they would have wanted a faster horse.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#104

Post by dotancohen »

kasey_junk wrote: post #101
herval wrote: post #76

Earlier quoted context omitted.

How does one “fix hallucinations” on an LLM? Isn’t hallucinating pretty much all it does?

Coding agents have shown how. You filter the output against something that can tell the llm when it’s hallucinating. The hard part is identifying those filter functions outside of the code domain.

It's called a RAG, and it's getting very well developed for some niche use cases such as legal, medical, etc. I've been personally working on one for mental health, and please don't let anybody tell you that they're using an LLM as a mental health counselor. I've been working on it for a year and a half, and if we get it to production ready in the next year and a half I will be surprised. In keeping up with the field, I don't think anybody else is any closer than we are.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#105

Post by dotancohen »

fexelein wrote: post #54
amelius wrote: post #50

Earlier quoted context omitted.

> I'd love to hook my development tools into a fully-local LLM. Karpathy said in his recent talk, on the topic of AI developer-assistants: don't bother with less capable models. So ... using an rpi is probably not what you want.

I’m having a lot of fun using less capable versions of models on my local PC, integrated as a code assistant. There still is real value there, but especially room for improvements. I envision us all running specialized lightweight LLMs locally/on-device at some point.

I'd love to hear more about what you're running, and on what hardware. Also, what is your use case? Thanks!

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#106

Post by dotancohen »

giancarlostoro wrote: post #67
behnamoh wrote: post #20

Everything runs on a π if you quantize it enough! I'm curious about the applications though. Do people randomly buy 4xRPi5s that they can now dedicate to running LLMs?

Sometimes you buy a pi for one project start on it buy another for a different project, before you know it none are complete and you have ten Raspberry Pis lying around across various generations. ;)

Arduino hobbist, same issue.

Though I must admit to first noticing the trend decades before discovering Arduino when I looked at the stack of 289, 302, and 351W intake manifolds on my shelf and realised that I need the width of the 351W manifold but the fuel injection of the 302. Some things just never change.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#107

Post by dotancohen »

piecerough wrote: post #90
behnamoh wrote: post #20

Everything runs on a π if you quantize it enough! I'm curious about the applications though. Do people randomly buy 4xRPi5s that they can now dedicate to running LLMs?

"quantize enough" though at what quality?

Quantity has a quality all its own.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#109

Post by codedokode »

trebligdivad wrote: post #93
dingdingdang wrote: post #4

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…

On my (single) AMD 3950x running entirely in CPU (llama -t32 -dev none), I was getting 14 tokens/s running Qwen3-Coder-30B-A3B-Instruct-IQ4_NL.gguf last night. Which is the best I've had out of a model that doesn't feel stupid.

How much RAM it is using by the way? I see 30B, but without knowing precision it is unclear how much memory one needs.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#110

Post by codedokode »

evilduck wrote: post #78
croes wrote: post #73

Earlier quoted context omitted.

What about AMD Ryzen AI Max+ 395 Mini PCs with upto 128GB unified memory?

Their memory bandwidth is the problem. 256 GB/s is really, really slow for LLMs. Seems like at the consumer hardware level you just have to pick your poison or what one factor you care about most. Macs with a Max or Ultra chip can have good memory bandwidth but low compute, but also ultra low power consumption. Discrete GPUs have great compute and bandwidth but low to middling VRAM, and high costs and power consumpti…

But for matrix multiplication, isn't compute more important, as there are N³ multiplications but just N² numbers in a matrix?

Also I don't think power consumption is important for AI. Typically you do AI at home or in the office where there is lot of electricity.

Post reply on HN