Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

github.com

Post reply on HN
Print view View original

91–100 of 171 posts

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#91

Post by littlestymaar »

amelius wrote: post #50
ryukoposting wrote: post #25

Earlier quoted context omitted.

I'd love to hook my development tools into a fully-local LLM. The question is context window and cost. If the context window isn't big enough, it won't be helpful for me. I'm not gonna drop $500 on RPis unless I know it'll be worth the money. I could try getting my employer to pay for it, but I'll probably have a much easier time convincing them to pay for Claude or whatever.

> I'd love to hook my development tools into a fully-local LLM. Karpathy said in his recent talk, on the topic of AI developer-assistants: don't bother with less capable models. So ... using an rpi is probably not what you want.

> Karpathy said in his recent talk, on the topic of AI developer-assistants: don't bother with less capable models.

Interesting because he also said the future is small "cognitive core" models:

> a few billion param model that maximally sacrifices encyclopedic knowledge for capability. It lives always-on and by default on every computer as the kernel of LLM personal computing.

https://xcancel.com/karpathy/status/1938626382248149433#m

In which case, a raspberry Pi sounds like what you need.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#92

Post by Twirrim »

tonyhart7 wrote: post #14
Twirrim wrote: post #11

Earlier quoted context omitted.

It's not unrealistically pessimistic. We're already seeing research showing the negative effects, as well as seeing routine psychosis stories. Think about the ways that LLMs interact. The constant barrage of positive responses "brilliant observation" etc. That's not a healthy input to your mental feedback loop. We all need responses that are grounded in reality, just like you'd get from other human beings. Think abou…

Yes this is flaw on we train them, we must rethink on how rewards reinforced learning works but that doesn't mean its not fixable, that doesn't mean progress must stop if the earliest inventor of plane think like you, human would never conquer skies we are in explosive growth that many brightest mind in planet get recruited to solve this problem, in fact I would be baffled if we didn't solve this by the end of year i…

Wow. That's... one hell of a leap you're making.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#93

Post by trebligdivad »

dingdingdang wrote: post #4

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…

On my (single) AMD 3950x running entirely in CPU (llama -t32 -dev none), I was getting 14 tokens/s running Qwen3-Coder-30B-A3B-Instruct-IQ4_NL.gguf last night. Which is the best I've had out of a model that doesn't feel stupid.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#94

Post by MengerSponge »

mycall wrote: post #61
ivape wrote: post #43

Earlier quoted context omitted.

People don’t know what they want yet, you have to show it to them. Getting the hardware out is part of it, but you are right, we’re missing the killer apps at the moment. The very need for privacy with AI will make personal hardware important no matter what.

Two main factors are holding back the "killer app" for AI. Fix hallucinations and make agents more deterministic. Once these are in place, people will love AI when it can make them money somehow.

Other than that, Mrs. Lincoln, how was the Agentic AI?

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#95

Post by mrbonner »

giancarlostoro wrote: post #65
rthnbgrredf wrote: post #8

Earlier quoted context omitted.

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

You dont even need Asahi, you can run comfy on it but I recommend the Draw Things app, it just works and holds your hand a LOT. I am able to run a few models locally, the underlying app is open source.

I used Draw Thing after fighting with comfyui.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#96

Post by pseudosavant »

throaway920181 wrote: post #72
ryukoposting wrote: post #25

Earlier quoted context omitted.

I'd love to hook my development tools into a fully-local LLM. The question is context window and cost. If the context window isn't big enough, it won't be helpful for me. I'm not gonna drop $500 on RPis unless I know it'll be worth the money. I could try getting my employer to pay for it, but I'll probably have a much easier time convincing them to pay for Claude or whatever.

It's sad that Pis are now so overpriced. They used to be fun little tinker boards that were semi-cheap.

The Raspberry Pi 2 Zero is as fast as a Pi 3, way smaller, and only costs $13 I think.

The high end Pis aren’t $25 though.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#97

Post by Aurornis »

rthnbgrredf wrote: post #8
dingdingdang wrote: post #4

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

> and install Asahi Linux for tinkering.

I would recommend sticking to macOS if compatibility and performance are the goal.

Asahi is an amazing accomplishment, but running native optimized macOS software including MLX acceleration is the way to go unless you’re dead-set on using Linux and willing to deal with the tradeoffs.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#98

Post by Aurornis »

wkat4242 wrote: post #84
ivape wrote: post #21

Earlier quoted context omitted.

It just came to my attention that the 2021 M1 Max 64gb is less than $1500 used. That’s 64gb of unified memory at regular laptop prices, so I think people will be well equipped with AI laptops rather soon. Apple really is #2 and probably could be #1 in AI consumer hardware.

M1 doesn't exactly have stellar memory bandwidth for this day and age though

M1 Max with 64GB has 400GB/s memory bandwidth.

You have to get into the highest 16-core M4 Max configurations to begin pulling away from that number.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#99

Post by Aurornis »

supportengineer wrote: post #40
echelon wrote: post #3

This is really impressive. If we can get this down to a single Raspberry Pi, then we have crazy embedded toys and tools. Locally, at the edge, with no internet connection. Kids will be growing up with toys that talk to them and remember their stories. We're living in the sci-fi future. This was unthinkable ten years ago.

[flagged]

Parent here. Kids have a lot of time and do a lot of different things. Some times it rains or snows or we’re home sick. Kids can (and will) do a lot of different things and it’s good to have options.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#100

Post by estimator7292 »

wkat4242 wrote: post #83
estimator7292 wrote: post #48

Earlier quoted context omitted.

99.9% of enthusiast/hobbyist clusters like this are exclusively used for blinkenlights

Blinkenlights are an admirable pursuit

That wasn't a judgement! I filled my homelab rack server with mechanical drives so I can get clicky noises along with the blinky lights
Post reply on HN