Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A 30B Qwen model walks into a Raspberry Pi and runs in real time

byteshape.com

Post reply on HN
Print view View original

21–30 of 139 posts

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#21

Post by jake_wls »

Alex2037 wrote: post #19
Imustaskforhelp wrote: post #12

Earlier quoted context omitted.

https://chatgpt.com/share/695d9ac2-c314-8011-8938-b0d7de7059... You can paste any article and chatgpt (took the most laymen AI thing) and just writing summarize this article https://byteshape.com/blogs/Qwen3-30B-A3B-Instruct-2507/ can give you insights about it. Although I am all for freedom, one forgets that this is one of the few places left on internet where discussions feel meaningful and I am not judging you if…

>we should read other people's articles sure, and reading a LLM summary allows one to decide whether the full article is worth reading or not.

[flagged]

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#22

Post by Imustaskforhelp »

6510 wrote: post #18
Imustaskforhelp wrote: post #12

Earlier quoted context omitted.

https://chatgpt.com/share/695d9ac2-c314-8011-8938-b0d7de7059... You can paste any article and chatgpt (took the most laymen AI thing) and just writing summarize this article https://byteshape.com/blogs/Qwen3-30B-A3B-Instruct-2507/ can give you insights about it. Although I am all for freedom, one forgets that this is one of the few places left on internet where discussions feel meaningful and I am not judging you if…

I'm not entirely sure but I think if the file name ends with .user.js like HN%20ChatGPT%20Summarize.user.js it will prompt to install when opening the raw file. haha, like so works too https://raw.githubusercontent.com/SerJaimeLannister/tampermo...

Alright so I did change the name of the file from HN ChatGPT Summarize.js to hn-summarize-ai.user.js

Is this what you are talking about? If you need any cooperation from my side lemme know, I don't know too much about tampermonkey but I end up using it for my mini scripts because its way much easier to deal with compared to building pure extensions themselves and these have their own editors as well so I just copy paste for a faster way to prototype with stuff like this

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#23

Post by Imustaskforhelp »

Alex2037 wrote: post #19
Imustaskforhelp wrote: post #12

Earlier quoted context omitted.

https://chatgpt.com/share/695d9ac2-c314-8011-8938-b0d7de7059... You can paste any article and chatgpt (took the most laymen AI thing) and just writing summarize this article https://byteshape.com/blogs/Qwen3-30B-A3B-Instruct-2507/ can give you insights about it. Although I am all for freedom, one forgets that this is one of the few places left on internet where discussions feel meaningful and I am not judging you if…

>we should read other people's articles sure, and reading a LLM summary allows one to decide whether the full article is worth reading or not.

Fair I guess, I think I myself might use it when sometimes the articles are more dense than my liking perhaps. As I said, I just built it out of curiosity but also a solution to their problem because I didnt like the idea of having an AI generated summary in the comments.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#24

Post by jdright »

jareds wrote: post #20

Is there a good place for easy comparisons of different models? I know gpt-oss-20b and gpt-oss-120b have different numbers of parameters, but don't know what this means in practice. All my experience with AI has been with larger models like Gemini and GPT. I'm interested in running models on my own hardware but don't know how small I can go and still get useful output both for simple things like fixing spelling and g…

https://swe-rebench.com/

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#25

Post by throwaway7783 »

jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

And toys

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#26

Post by LargoLasskhyfv »

geerlingguy wrote: post #8

I've just tried replicating this on my Pi 5 16GB, running the latest llama.cpp... and it segfaults: ./build/bin/llama-cli -m "models/Qwen3-30B-A3B-Instruct-2507-Q3_K_S-2.70bpw.gguf" -e --no-mmap -t 4 ... Loading model... -ggml_aligned_malloc: insufficient memory (attempted to allocate 24576.00 MB) ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 25769803776 alloc_tensor_range: failed to al…

Have you tried anything with https://codeberg.org/ikawrakow/illama

https://github.com/ikawrakow/ik_llama.cpp and their 4Bit-quants?

Or maybe even Microsofts Bitnet? https://github.com/microsoft/BitNet

https://github.com/ikawrakow/ik_llama.cpp/pull/337

https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf ?

That would be an interesting comparison for running local LLMs on such low-end/edge-devices. Or common office machines with only iGPU.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#27

Post by nickthegreek »

protocolture wrote: post #7
jmward01 wrote: post #6

There is a huge market segment waiting here. At least I think there is. Well, at least people like me want this. Ok, tens of dollars can be made at least. It is just missing a critical tipping point. Basically, I want an alexa like device for the home backed by local inference and storage with some standardized components identified: - the interactive devices - all the alexa/google/apple devices out there are this in…

Keen for this also. Been having issues getting a smooth voice experience from HA to ChatGPT. I dont like the whole wakeword concept for the receiver either. I think theres work to be done on the whole stack.

you can use a physical button instead of wakeword.

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#28

Post by sdenton4 »

6510 wrote: post #16
protocolture wrote: post #7

Earlier quoted context omitted.

Keen for this also. Been having issues getting a smooth voice experience from HA to ChatGPT. I dont like the whole wakeword concept for the receiver either. I think theres work to be done on the whole stack.

It should participate in all conversations, take initiative and experiment.

"Hey, hey, are you still asleep? Using spare cycles, I have designed an optimal recipe for mashed potatoes, as you mentioned ten days ago. I need you to go get some potatoes."

Re: A 30B Qwen model walks into a Raspberry Pi and runs in real time

#30

Post by Aurornis »

CSSer wrote: post #3
yjftsjthsd-h wrote: post #2

In case anyone else clicked in wondering what counts as "real time" for this: > On a Pi 5 (16GB), Q3_K_S-2.70bpw [KQ-2] hits 8.03 TPS at 2.70 BPW and maintains 94.18% of BF16 quality. And they talk about other hardware and details. But that's the expanded version of the headline claim.

Someone should make a version of the Hacker News homepage that is just LLM extracts of key article details like this.

If you read a lot of comment sections, there are bot accounts showing up on LLM that try to do this constantly.

Their output is not great so they get downvoted and spotted quickly.

Post reply on HN