Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 30B Qwen model walks into a Raspberry Pi and runs in real time

byteshape.com

Post reply on HN
Print view View original

81–90 of 139 posts

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#81

Post by kouteiheika »

tgv wrote: post #71
kouteiheika wrote: post #46

Earlier quoted context omitted.

> What does it mean that only 3B parameters are active at a time? In a nutshell: LLMs generate tokens one at a time. "only 3B parameters active a a time" means that for each of those tokens only 3B parameters need to be fetched from memory, instead of all of them (30B).

Then I don't understand why it would matter. Or does it really mean that for each input token 10% of the total network runs, and then another 10% for the next token, rather than running each 10 batches of 10% for each token? If so, any idea or pointer to how the selection works?

Yes, for each token only, say, 10% of the weights are necessary, so you don't have to fetch the remaining 90% from memory, which makes inference much faster (if you're memory bound; if you're doing single batch inference then you're certainly memory bound).

As to how the selection works - each mixture-of-experts layer in the netwosk has essentially a small subnetwork called a "router" which looks at the input and calculates the scores for each expert; then the best scoring experts are picked and the inputs are only routed to them.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#82

Post by nl »

I've been super impressed by qwen3:0.6b (yes, 0.6B) running in Ollama.

If you have very specific, constrained tasks it can do quite a lot. It's not perfect though.

https://tools.nicklothian.com/llm_comparator.html?gist=fcae9... is an example conversation where I took OpenAI's "Natural language to SQL" prompt[1], send it to Ollama:qwen3:0.6b and the asked Gemini Flash 3 to compare what qwen3:0.6b did vs what Flash did.

Flash was clearly correct, but the qwen3:0.6b errors are interesting in themselves.

[1] https://platform.openai.com/docs/examples/default-sql-transl...

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#83

Post by sallveburrpi »

mcny wrote: post #78
Normal_gaussian wrote: post #17

Earlier quoted context omitted.

No, there isn't a plug and play one yet, but I've have great success with Home Assistant and the Home Assistant Voice Preview edition and its goal is pretty much to get rid of Alexa. I'd imagine you'd have a bunch of cheap ones in the house that are all WiFi + Mic + Speakers, streaming back to your actual voice processing box (which would cost a wee bit more, but also have local access to all the data it needs). You…

And if it is plugged in to the wall, I'd be tempted to add a touch screen display and a camera just in case. But really my use case is as simple as 1. Wake word, what time is it in ____ 2. Wake word, how is the weather in ____ 3. Wake word, will it rain/snow/?? in _____ today / tomorrow / ?? 4. Wake word, what is ______ 5. Wake word, when is the next new moon / full moon? 6. Wake word, when is sunrise / sunset? And s…

So you need a clock maybe? Plus something like wttr.in

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#84

Post by quietsegfault »

sdenton4 wrote: post #28
6510 wrote: post #16

Earlier quoted context omitted.

It should participate in all conversations, take initiative and experiment.

"Hey, hey, are you still asleep? Using spare cycles, I have designed an optimal recipe for mashed potatoes, as you mentioned ten days ago. I need you to go get some potatoes."

This sounds a lot like gptars. I want a little gptars tearing around my house.

https://youtube.com/shorts/e2t0RxX4b54

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#85

Post by BoxOfRain »

matusp wrote: post #55
jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

The sota chatbots are getting more and more functionality that is not just LLM inference. They can search the web, process files, integrate with other apps. I think that's why most people will consider local LLMs to be insufficient very soon.

Nah I disagree, tool calling isn't that difficult. I've got my own Cats Effect based model orchestration project I'm working on, and while it's not 100% yet I can do web browse, web search, memory search (this one is cool), and others on my own hardware.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#86

Post by Aurornis »

nl wrote: post #82

I've been super impressed by qwen3:0.6b (yes, 0.6B) running in Ollama. If you have very specific, constrained tasks it can do quite a lot. It's not perfect though. https://tools.nicklothian.com/llm_comparator.html?gist=fcae9... is an example conversation where I took OpenAI's "Natural language to SQL" prompt[1], send it to Ollama:qwen3:0.6b and the asked Gemini Flash 3 to compare what qwen3:0.6b did vs what Flash did…

I’ve experimented with several of the really small models. It’s impressive that they can produce anything at all, but in my experience the output is basically useless for anything of value.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#87

Post by Aurornis »

anonzzzies wrote: post #29

We need custom inference chips at scale for this imho. Every computer (whatever formfactor/board) should have an inference unit on it so at least inference is efficient and fast and can be offloaded while the cpu is doing something else.

The bottleneck in common PC hardware is mostly memory bandwidth. Offloading the computation part to a different chip wouldn’t help if memory access is the bottleneck.

There have been a lot of boards and chips for years with dedicated compute hardware, but they’re only so useful for these LLM models that require huge memory bandwidth.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#88

Post by nl »

Aurornis wrote: post #86
nl wrote: post #82

I've been super impressed by qwen3:0.6b (yes, 0.6B) running in Ollama. If you have very specific, constrained tasks it can do quite a lot. It's not perfect though. https://tools.nicklothian.com/llm_comparator.html?gist=fcae9... is an example conversation where I took OpenAI's "Natural language to SQL" prompt[1], send it to Ollama:qwen3:0.6b and the asked Gemini Flash 3 to compare what qwen3:0.6b did vs what Flash did…

I’ve experimented with several of the really small models. It’s impressive that they can produce anything at all, but in my experience the output is basically useless for anything of value.

Yes, I thought that too! But qwen3:0.6b (and to some extent gemma 1b) has made me reevaluate.

They still aren't useful like large LLMs, but for things like summarization, and other tasks where you can give them structure but want the sheen of natural language they are much better than things like the Phi series were.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#89

Post by Aurornis »

geerlingguy wrote: post #10
batch12 wrote: post #9

Earlier quoted context omitted.

Could they have added some swap?

No, just updated the parent comment, I added -c 4096 to cut down the context size, and now the model loads. I'm able to get 6-7 tokens/sec generation with 10-11 tokens/sec prompt processing with their model. Seems quite good, actually—much more useful than llama 3.2:3b, which has comparable performance on this Pi.

> I added -c 4096 to cut down the context size

That’s a pretty big caveat. In my experience, using a small context size is only okay for very short answers and questions. The output looks coherent until you try to use it for anything, then it turns into the classic LLM babble that looks like words are being put into a coherent order but the sum total of the output is just rambling.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#90

Post by nallic »

geerlingguy wrote: post #10
batch12 wrote: post #9

Earlier quoted context omitted.

Could they have added some swap?

No, just updated the parent comment, I added -c 4096 to cut down the context size, and now the model loads. I'm able to get 6-7 tokens/sec generation with 10-11 tokens/sec prompt processing with their model. Seems quite good, actually—much more useful than llama 3.2:3b, which has comparable performance on this Pi.

for some reason I only get 3-4 tokens/sec. I checked the CPU does not throttle or anything.
Post reply on HN