Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

github.com

Post reply on HN
Print view View original

141–150 of 171 posts

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#141

Post by seanmcdirmid »

ivape wrote: post #21
rthnbgrredf wrote: post #8

Earlier quoted context omitted.

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering. Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

It just came to my attention that the 2021 M1 Max 64gb is less than $1500 used. That’s 64gb of unified memory at regular laptop prices, so I think people will be well equipped with AI laptops rather soon. Apple really is #2 and probably could be #1 in AI consumer hardware.

I recently got an M3 Max with 64g (the higher spec max) and ts been a lot of fun playing with local models. It cost around $3k though even refurbished.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#142

Post by 1gn15 »

echelon wrote: post #3

This is really impressive. If we can get this down to a single Raspberry Pi, then we have crazy embedded toys and tools. Locally, at the edge, with no internet connection. Kids will be growing up with toys that talk to them and remember their stories. We're living in the sci-fi future. This was unthinkable ten years ago.

This is indeed incredibly sci fi. I still remember my ChatGPT moment, when I realized I could actually talk to a computer. And now it can run fully on an RPi, just as if the RPi itself has become intelligent and articulate! Very cool.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#144

Post by tonyhart7 »

yepitwas wrote: post #75
tonyhart7 wrote: post #22

Earlier quoted context omitted.

are you going to give up knife too because some people use it for crime????

Do you think I am somehow bound to answer yes to this question? If so, why do you think that?

You would not admit that because you have an ego that would expose your flawed logic

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#145

Post by dingdingdang »

herval wrote: post #76
mycall wrote: post #61

Earlier quoted context omitted.

Two main factors are holding back the "killer app" for AI. Fix hallucinations and make agents more deterministic. Once these are in place, people will love AI when it can make them money somehow.

How does one “fix hallucinations” on an LLM? Isn’t hallucinating pretty much all it does?

No no, not at all, see: https://openai.com/index/why-language-models-hallucinate/ which was recently featured on the frontpage - excellent clean take on how to fix the issue (they already got a long way with gpt-5-thinking-mini). I liked this bit for clear outline of the issue:

´´´Think about it like a multiple-choice test. If you do not know the answer but take a wild guess, you might get lucky and be right. Leaving it blank guarantees a zero. In the same way, when models are graded only on accuracy, the percentage of questions they get exactly right, they are encouraged to guess rather than say “I don’t know.”

As another example, suppose a language model is asked for someone’s birthday but doesn’t know. If it guesses “September 10,” it has a 1-in-365 chance of being right. Saying “I don’t know” guarantees zero points. Over thousands of test questions, the guessing model ends up looking better on scoreboards than a careful model that admits uncertainty."´´´´

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#147

Post by estimator7292 »

ivape wrote: post #43
jeroenhd wrote: post #29

Earlier quoted context omitted.

Apple is leagues ahead of Microsoft with the whole AI PC thing and so far it has yet to mean anything. I don't think consumers care at all about running AI, let alone running AI locally. I'd try the whole AI thing on my work Macbook but Apple's built-in AI stuff isn't available in my language, so perhaps that's also why I haven't heard anybody mention it.

People don’t know what they want yet, you have to show it to them. Getting the hardware out is part of it, but you are right, we’re missing the killer apps at the moment. The very need for privacy with AI will make personal hardware important no matter what.

We've shown people so many times and so forcefully that they're now actively complaining about it. It's a meme.

The problem isn't getting your Killer A I App in front of eyeballs. The problem is showing something useful or necessary or wanted. AI has not yet offered the common person anything they want or need! The people have seen what you want to show them, they've been forced to try it, over and over. There is nobody who interacts with the internet who has not been forced to use AI tools.

And yet still nobody wants it. Do you think that they'll love AI more if we force them to use it more?

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#148

Post by SillyUsername »

wkat4242 wrote: post #85
SillyUsername wrote: post #24

Earlier quoted context omitted.

The irony of this is that Gen-Z have been mollycoddled with praise by their parents and modern life, we give medals for participation, or runners up prizes for losing. We tell people when they've failed at something they did their best and that's what matters. We validate their upset feelings if they're insulted by free speech that goes against their beliefs. This is exactly what is happening with sycophantic LLMs, t…

You sound very old man yelling at cloud. And the winner takes all is so American. And no discrimination against lgbt etc under the guise of free speech is not ok.

Well you're wrong on all accounts of the veiled insults.

Also, I've not stated LGBT, this has nothing to do with it, it's weird you'd even mention it.

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#149

Post by fexelein »

dotancohen wrote: post #105
fexelein wrote: post #54

Earlier quoted context omitted.

I’m having a lot of fun using less capable versions of models on my local PC, integrated as a code assistant. There still is real value there, but especially room for improvements. I envision us all running specialized lightweight LLMs locally/on-device at some point.

I'd love to hear more about what you're running, and on what hardware. Also, what is your use case? Thanks!

So I am running Ollama on Windows using an 10700k and 3080ti. I'm using models like Qwen3-coder (4/8b) and 2.5-coder 15b, Llama 3 instruct, etc. These models are very fast on my machine (~25-100 tokens per second depending on model)

My use case is custom software that I build and host that leverages LLMs for example for domotica where I use my Apple watch shortcuts to issue commands. I also created a VS2022 extension called Bropilot to replace Copilot with my locally hosted LLMs. Currently looking at fine tuning these type of models for work where I work in finance as a senior dev

Re: Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

#150

Post by ThatPlayer »

rao-v wrote: post #123

Nice! Cheap RK3588 boards come with 15GB of LPDDR5 RAM these days and have significantly better performance than the Pi 5 (and often are cheaper). I get 8.2 tokens per second on a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB). I need to try two of them in parallel ... should be significantly faster than this even at Q6.

Is that using the NPU on that board? I know it's possible to use those too.
Post reply on HN