Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 30B Qwen model walks into a Raspberry Pi and runs in real time

byteshape.com

Post reply on HN
Print view View original

71–80 of 139 posts

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#71

Post by tgv »

kouteiheika wrote: post #46
throwaway894345 wrote: post #41

Earlier quoted context omitted.

What does it mean that only 3B parameters are active at a time? Also any indication of whether this was purely CPU or if it’s using the Pi’s GPU?

> What does it mean that only 3B parameters are active at a time? In a nutshell: LLMs generate tokens one at a time. "only 3B parameters active a a time" means that for each of those tokens only 3B parameters need to be fetched from memory, instead of all of them (30B).

Then I don't understand why it would matter. Or does it really mean that for each input token 10% of the total network runs, and then another 10% for the next token, rather than running each 10 batches of 10% for each token? If so, any idea or pointer to how the selection works?

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#73

Post by fennecbutt »

protocolture wrote: post #7
jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

Keen for this also. Been having issues getting a smooth voice experience from HA to ChatGPT. I dont like the whole wakeword concept for the receiver either. I think theres work to be done on the whole stack.

What's wrong with the wakeword stuff?

Great timing as I was looking into it yesterday as was thinking about writing my own set of agents to run house stuff. I don't want to spent loads of time on voice interaction so HA wakeword stuff would've been useful. If not I'll bypass HA for voice and really only use HA via mcp.

I can do fw dev for micros...but omg do I not want to spend the time looking thru a datasheet and getting something to run efficiently myself these days.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#74

Post by baq »

oliwary wrote: post #72

Does anyone have any use-cases for long-running, interesting tasks that do not require perfect accuracy? Seems like this would be the sweet spot for running local models on low-powered hardware.

looking at the dump of your home assistant server and sensor data and saying 'hmm that's interesting' if it notices something interesting

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#75

Post by mkul »

jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

I've just started using it but I'd recommend https://github.com/steipete/clawdis, you need to set it up a bit but it's really cool to just be able to do things on the go by just texting an assistant. You can see all the different ways people are using it @clawdbot on twitter.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#76

Post by mr_mitm »

mkul wrote: post #75
jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

I've just started using it but I'd recommend https://github.com/steipete/clawdis , you need to set it up a bit but it's really cool to just be able to do things on the go by just texting an assistant. You can see all the different ways people are using it @clawdbot on twitter.

Can you give us some highlights on how this is helpful in your day-to-day life for those of us who aren't on twitter?

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#77

Post by Latitude7973 »

ukuina wrote: post #61
grosswait wrote: post #36

Earlier quoted context omitted.

Not sure if it is still updating https://hackyournews.com/

Thanks for pointing this out, https://hackyournews.com should be up and running again!

Is this your project? It would be great to bolster it with links to comment sections and the current points tally.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#78

Post by mcny »

Normal_gaussian wrote: post #17
jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

No, there isn't a plug and play one yet, but I've have great success with Home Assistant and the Home Assistant Voice Preview edition and its goal is pretty much to get rid of Alexa. I'd imagine you'd have a bunch of cheap ones in the house that are all WiFi + Mic + Speakers, streaming back to your actual voice processing box (which would cost a wee bit more, but also have local access to all the data it needs). You…

And if it is plugged in to the wall, I'd be tempted to add a touch screen display and a camera just in case.

But really my use case is as simple as

1. Wake word, what time is it in ____

2. Wake word, how is the weather in ____

3. Wake word, will it rain/snow/?? in _____ today / tomorrow / ??

4. Wake word, what is ______

5. Wake word, when is the next new moon / full moon?

6. Wake word, when is sunrise / sunset?

And something similar like that

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#80

Post by nl »

ekidd wrote: post #34
jareds wrote: post #20

Is there a good place for easy comparisons of different models? I know gpt-oss-20b and gpt-oss-120b have different numbers of parameters, but don't know what this means in practice. All my experience with AI has been with larger models like Gemini and GPT. I'm interested in running models on my own hardware but don't know how small I can go and still get useful output both for simple things like fixing spelling and g…

One easy way to test different models is purchase $20 worth of tokens from one of the Open Router-like sites. This will let you asks tons of questions and try out lots of models. Realistically, the biggest models you can run at a reasonable price right now are quantized versions of things like the Qwen3 30B A3B family. A 4-bit quantized version fits in roughly 15GB of RAM. This will run very nicely on something like…

I wrote https://tools.nicklothian.com/llm_comparator.html so you can compare different models.

OpenRouter gives you $10 credit when you sign up - stick your API key in and compare as many models as you want. It's all browser local storage.

Post reply on HN