Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

github.com

Post reply on HN
Print view View original

21–30 of 171 posts

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#21

Post by ivape »

rthnbgrredf wrote: post #8
dingdingdang wrote: post #4

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

It just came to my attention that the 2021 M1 Max 64gb is less than $1500 used. That’s 64gb of unified memory at regular laptop prices, so I think people will be well equipped with AI laptops rather soon.

Apple really is #2 and probably could be #1 in AI consumer hardware.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#22

Post by tonyhart7 »

yepitwas wrote: post #16
tonyhart7 wrote: post #9

Earlier quoted context omitted.

this is very pessimistic take there are lot of bad people on internet too, does that make internet is a mistake ??? Noo, the people are not the tool

> there are lot of bad people on internet too, does that make internet is a mistake ??? Yes. People write and say “the Internet was a mistake” all the time, and some are joking, but a lot of us aren’t.

are you going to give up knife too because some people use it for crime????

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#23

Post by j45 »

dingdingdang wrote: post #4

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…

Connect a gpu into it with an eGPU chassis and you're running one way or the other.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#24

Post by SillyUsername »

Twirrim wrote: post #11
tonyhart7 wrote: post #9

Earlier quoted context omitted.

this is very pessimistic take there are lot of bad people on internet too, does that make internet is a mistake ??? Noo, the people are not the tool

It's not unrealistically pessimistic. We're already seeing research showing the negative effects, as well as seeing routine psychosis stories. Think about the ways that LLMs interact. The constant barrage of positive responses "brilliant observation" etc. That's not a healthy input to your mental feedback loop. We all need responses that are grounded in reality, just like you'd get from other human beings. Think abou…

The irony of this is that Gen-Z have been mollycoddled with praise by their parents and modern life, we give medals for participation, or runners up prizes for losing. We tell people when they've failed at something they did their best and that's what matters. We validate their upset feelings if they're insulted by free speech that goes against their beliefs.

This is exactly what is happening with sycophantic LLMs, to a greater extent, but now it's affecting other generations, not just Gen-Z.

Perhaps it's time to rollback this behaviour in the human population too, and no I'm not talking reinstating discipline and old Boomer/Gen-X practices, I'm meaning that we need to allow more failure and criticism without comfort and positive reinforcement.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#25

Post by ryukoposting »

behnamoh wrote: post #20

Everything runs on a π if you quantize it enough! I'm curious about the applications though. Do people randomly buy 4xRPi5s that they can now dedicate to running LLMs?

I'd love to hook my development tools into a fully-local LLM. The question is context window and cost. If the context window isn't big enough, it won't be helpful for me. I'm not gonna drop $500 on RPis unless I know it'll be worth the money. I could try getting my employer to pay for it, but I'll probably have a much easier time convincing them to pay for Claude or whatever.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#27

Post by lostmsu »

hidelooktropic wrote: post #10
mehdibl wrote: post #5

[flagged]

13/s is not slow. Q4 is not bad. The models that run on phones are never 30B or anywhere close to that.

It is very slow and totally unimpressive. 5060Ti ($430 new) would do over 60, even more in batched mode. 4x RPi 5 are $550 new.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#28

Post by jibbers »

rthnbgrredf wrote: post #8
dingdingdang wrote: post #4

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with…

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

Get an Apple Silicon MacBook with a broken screen and it’s an even better deal.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#29

Post by jeroenhd »

ivape wrote: post #21
rthnbgrredf wrote: post #8

Earlier quoted context omitted.

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

It just came to my attention that the 2021 M1 Max 64gb is less than $1500 used. That’s 64gb of unified memory at regular laptop prices, so I think people will be well equipped with AI laptops rather soon. Apple really is #2 and probably could be #1 in AI consumer hardware.

Apple is leagues ahead of Microsoft with the whole AI PC thing and so far it has yet to mean anything. I don't think consumers care at all about running AI, let alone running AI locally.

I'd try the whole AI thing on my work Macbook but Apple's built-in AI stuff isn't available in my language, so perhaps that's also why I haven't heard anybody mention it.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#30

Post by magicalhippo »

kosolam wrote: post #15

How is this technically done? How does it split the query and aggregates the results?

From the readme:

More devices mean faster performance, leveraging tensor parallelism and high-speed synchronization over Ethernet.

The maximum number of nodes is equal to the number of KV heads in the model #70.

I found this[1] article nice for an overview of the parallelism modes.

[1]: https://medium.com/@chenhao511132/parallelism-in-llm-inferen...

Post reply on HN