Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 30B Qwen model walks into a Raspberry Pi and runs in real time

byteshape.com

Post reply on HN
Print view View original

31–40 of 139 posts

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#31

Post by bigyabai »

Alex2037 wrote: post #19
Imustaskforhelp wrote: post #12

Earlier quoted context omitted.

https://chatgpt.com/share/695d9ac2-c314-8011-8938-b0d7de7059... You can paste any article and chatgpt (took the most laymen AI thing) and just writing summarize this article https://byteshape.com/blogs/Qwen3-30B-A3B-Instruct-2507/ can give you insights about it. Although I am all for freedom, one forgets that this is one of the few places left on internet where discussions feel meaningful and I am not judging you if…

>we should read other people's articles sure, and reading a LLM summary allows one to decide whether the full article is worth reading or not.

Seems like a bad habit for media literacy.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#32

Post by fuzzer371 »

jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

And there never will be. You know why? Because the giant corporations can't suck up all your data and tailor advertisements to you. Why sell a good thing once, when you can sell crappy shovelware ridden with ads and a subscription service every month?

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#33

Post by syntaxing »

I feel like calling it a “30B” model is slightly disingenuous. It’s a 30B-A3B. So only 3B parameters is active at a given time. While still impressive nevertheless, being able to get 8T/s for a “A3B” compared to a dense 30B is very different.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#34

Post by ekidd »

jareds wrote: post #20

Is there a good place for easy comparisons of different models? I know gpt-oss-20b and gpt-oss-120b have different numbers of parameters, but don't know what this means in practice. All my experience with AI has been with larger models like Gemini and GPT. I'm interested in running models on my own hardware but don't know how small I can go and still get useful output both for simple things like fixing spelling and g…

One easy way to test different models is purchase $20 worth of tokens from one of the Open Router-like sites. This will let you asks tons of questions and try out lots of models.

Realistically, the biggest models you can run at a reasonable price right now are quantized versions of things like the Qwen3 30B A3B family. A 4-bit quantized version fits in roughly 15GB of RAM. This will run very nicely on something like an Nvidia 3090. But you can also use your regular RAM (though it will be slower).

These models aren't competitive with GPT 5 or Opus 4.5! But they're mostly all noticeably better than GPT-4o, some by quite a bit. Some of the 30B models will run as basic agentic coders.

There are also some great 4B to 8B models from various organizations that will fit on smaller systems. A 8B model, for example, can be a great translator.

(If you have a bunch of money and patience, you can also run something like GPT OSS 120B or GLM 4.5 Air locally.)

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#35

Post by cmrdporcupine »

ekidd wrote: post #34
jareds wrote: post #20

Is there a good place for easy comparisons of different models? I know gpt-oss-20b and gpt-oss-120b have different numbers of parameters, but don't know what this means in practice. All my experience with AI has been with larger models like Gemini and GPT. I'm interested in running models on my own hardware but don't know how small I can go and still get useful output both for simple things like fixing spelling and g…

One easy way to test different models is purchase $20 worth of tokens from one of the Open Router-like sites. This will let you asks tons of questions and try out lots of models. Realistically, the biggest models you can run at a reasonable price right now are quantized versions of things like the Qwen3 30B A3B family. A 4-bit quantized version fits in roughly 15GB of RAM. This will run very nicely on something like…

This is the answer. There's a half dozen sites that let you run these models by the token, and actually $20 is excessive. $5 will get you a long long way.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#36

Post by grosswait »

CSSer wrote: post #3
yjftsjthsd-h wrote: post #2

In case anyone else clicked in wondering what counts as "real time" for this: > On a Pi 5 (16GB), Q3_K_S-2.70bpw [KQ-2] hits 8.03 TPS at 2.70 BPW and maintains 94.18% of BF16 quality. And they talk about other hardware and details. But that's the expanded version of the headline claim.

Someone should make a version of the Hacker News homepage that is just LLM extracts of key article details like this.

Not sure if it is still updating https://hackyournews.com/

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#37

Post by jmward01 »

fuzzer371 wrote: post #32
jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

And there never will be. You know why? Because the giant corporations can't suck up all your data and tailor advertisements to you. Why sell a good thing once, when you can sell crappy shovelware ridden with ads and a subscription service every month?

Open source is amazing for this. Honestly, I suspect this is much simpler than the jellyfin ecosystem and other open source projects out there. Really, we are so close to this now it is just missing a few things like a good 'how to' that ties it all together and turns into the opensource repo that bundles things.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#38

Post by fouc »

anonzzzies wrote: post #29

We need custom inference chips at scale for this imho. Every computer (whatever formfactor/board) should have an inference unit on it so at least inference is efficient and fast and can be offloaded while the cpu is doing something else.

I can't believe this was downvoted. It makes a lot of sense that it would be highly useful to have mass custom inference chips.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#39

Post by tgtweak »

So basically the quantization in a byteshape model is per-tensor and can be variable and is an "average" in the final result? The results look good - curious why this isn't more prevalent! Would also love to better understand what factors into "accuracy" since there might be some nuance there depending on the measure.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#40

Post by geerlingguy »

geerlingguy wrote: post #14
layoric wrote: post #13

Earlier quoted context omitted.

Thanks for posting the performance numbers from your own validation. 6-7 tokens/sec is quite remarkable for the hardware.

Some more benchmarking, and with larger outputs (like writing an entire relatively complex TODO list app) it seems to go down to 4-6 tokens/s. Still impressive.

Decided to run an actual llama-bench run and let it go for the hour or two it needs. I'm posting my full results here (https://github.com/geerlingguy/ai-benchmarks/issues/47), but 8-10 t/s pp, and 7.99 t/s tg128, this is on a Pi 5 with no overclocking. Could probably increase the numbers slightly with an overclock.

You need to have a fan/heatsink to get that speed of course, it's maxing out the CPU for the entire time.

Post reply on HN