GPT-OSS-20B is only 11.2GB. Should fit in any 16GB machine with descent context without any quality degradation.
A 30B Qwen model walks into a Raspberry Pi and runs in real time
11–20 of 139 posts
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#12In case anyone else clicked in wondering what counts as "real time" for this: > On a Pi 5 (16GB), Q3_K_S-2.70bpw [KQ-2] hits 8.03 TPS at 2.70 BPW and maintains 94.18% of BF16 quality. And they talk about other hardware and details. But that's the expanded version of the headline claim.
Someone should make a version of the Hacker News homepage that is just LLM extracts of key article details like this.
https://chatgpt.com/share/695d9ac2-c314-8011-8938-b0d7de7059...
You can paste any article and chatgpt (took the most laymen AI thing) and just writing summarize this article https://byteshape.com/blogs/Qwen3-30B-A3B-Instruct-2507/
can give you insights about it.
Although I am all for freedom, one forgets that this is one of the few places left on internet where discussions feel meaningful and I am not judging you if you want AI but do it at your own discretion using chatbots.
If you want, you can even hack around a simple extension (tampermonkey etc.) where you can have a button which can do this for you if you really so desire.
Ended up being bored and asked chatgpt to do this but chatgpt is having something wrong, it got just blinking mode so I asked claude web (4.5 sonnet) to do it and I ended up building it with tampermonkey script.
Created the code. https://github.com/SerJaimeLannister/tampermonkey-hn-summari...
I was just writing this comment and I just got curious I guess so in the end ended up building it.
Although Edit: Thinking about it, I felt that we should read other people's articles as well. I just created this tool not out of endorsement of idea or anything but just curiosity or boredom but I think that we should probably read the articles themselves instead of asking chatgpt or LLM's about it.
There is this quote which I remembered right now
If something is worth talking/discussing about, its worth writing
If something is worth writing, then its worth reading.
Information that we write is fundamentally subjective (our writing style etc with our biases etc.), passing it through a black box which will try to homogenify all of it just feels like it misses the point.
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#13Earlier quoted context omitted.
Could they have added some swap?
No, just updated the parent comment, I added -c 4096 to cut down the context size, and now the model loads. I'm able to get 6-7 tokens/sec generation with 10-11 tokens/sec prompt processing with their model. Seems quite good, actually—much more useful than llama 3.2:3b, which has comparable performance on this Pi.
Thanks for posting the performance numbers from your own validation. 6-7 tokens/sec is quite remarkable for the hardware.
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#14Earlier quoted context omitted.
No, just updated the parent comment, I added -c 4096 to cut down the context size, and now the model loads. I'm able to get 6-7 tokens/sec generation with 10-11 tokens/sec prompt processing with their model. Seems quite good, actually—much more useful than llama 3.2:3b, which has comparable performance on this Pi.
Thanks for posting the performance numbers from your own validation. 6-7 tokens/sec is quite remarkable for the hardware.
Some more benchmarking, and with larger outputs (like writing an entire relatively complex TODO list app) it seems to go down to 4-6 tokens/s. Still impressive.
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#15In case anyone else clicked in wondering what counts as "real time" for this: > On a Pi 5 (16GB), Q3_K_S-2.70bpw [KQ-2] hits 8.03 TPS at 2.70 BPW and maintains 94.18% of BF16 quality. And they talk about other hardware and details. But that's the expanded version of the headline claim.
Someone should make a version of the Hacker News homepage that is just LLM extracts of key article details like this.
I mean, they didn't bury it far in the article, it's like a two second skim into it and it's labelled with a tl;dr. Not a bad idea in general but you don't even need it for this one.
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#16There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…
Keen for this also. Been having issues getting a smooth voice experience from HA to ChatGPT. I dont like the whole wakeword concept for the receiver either. I think theres work to be done on the whole stack.
It should participate in all conversations, take initiative and experiment.
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#17There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…
No, there isn't a plug and play one yet, but I've have great success with Home Assistant and the Home Assistant Voice Preview edition and its goal is pretty much to get rid of Alexa.
I'd imagine you'd have a bunch of cheap ones in the house that are all WiFi + Mic + Speakers, streaming back to your actual voice processing box (which would cost a wee bit more, but also have local access to all the data it needs).
You can see quite quickly that this becomes just another program running on a host, so if you use a slightly beefier machine and chuck a WiFi card in as well you've got your WiFi extenders.
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#18Earlier quoted context omitted.
Someone should make a version of the Hacker News homepage that is just LLM extracts of key article details like this.
https://chatgpt.com/share/695d9ac2-c314-8011-8938-b0d7de7059... You can paste any article and chatgpt (took the most laymen AI thing) and just writing summarize this article https://byteshape.com/blogs/Qwen3-30B-A3B-Instruct-2507/ can give you insights about it. Although I am all for freedom, one forgets that this is one of the few places left on internet where discussions feel meaningful and I am not judging you if…
I'm not entirely sure but I think if the file name ends with .user.js like HN%20ChatGPT%20Summarize.user.js it will prompt to install when opening the raw file.
haha, like so works too
https://raw.githubusercontent.com/SerJaimeLannister/tampermo...
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#19Earlier quoted context omitted.
Someone should make a version of the Hacker News homepage that is just LLM extracts of key article details like this.
https://chatgpt.com/share/695d9ac2-c314-8011-8938-b0d7de7059... You can paste any article and chatgpt (took the most laymen AI thing) and just writing summarize this article https://byteshape.com/blogs/Qwen3-30B-A3B-Instruct-2507/ can give you insights about it. Although I am all for freedom, one forgets that this is one of the few places left on internet where discussions feel meaningful and I am not judging you if…
>we should read other people's articles
sure, and reading a LLM summary allows one to decide whether the full article is worth reading or not.
Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time
#20Is there a good place for easy comparisons of different models? I know gpt-oss-20b and gpt-oss-120b have different numbers of parameters, but don't know what this means in practice. All my experience with AI has been with larger models like Gemini and GPT. I'm interested in running models on my own hardware but don't know how small I can go and still get useful output both for simple things like fixing spelling and grammar, as well as complex things like programming.