Pretty incredible performance for the footprint - really interested to see what could be done on slightly more powerful SBCs like some that have been mentioned in this thread.
Running a 28.9M parameter LLM on an $8 microcontroller
31–40 of 77 posts
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#32Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD. I think they got something like 10 _seconds per token_ (not tokens per second). EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri
So while SSD streaming is interesting I'm not sure it's exactly the same thing as the per-layer embedding that is being utilized in tandem with streaming here. To utilize per-layer embedding, it would have had to be trained that way, which GLM 5.2 was not.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#33Earlier quoted context omitted.
Fun and learning is a really good reason. It also reminds me of the damascene: trying to achieve something that doesn't feel possible, and working through all the extreme resource-constrained engineering limits. I think most of my embedded projects aren't that useful, but they've taught me a lot.
I'm not disagreeing with you, I think this goes beyond impractical, it's doomed from the get go. In my book, impractical means "I built a cuckoo wristwatch". Beyond impractical: "I built a cuckoo wristwatch but there was no room for a working mechanism".
I don't see the relationship between "fun and learning" and "(beyond) impractical".
I enter most of my learning project from the assumption that I could just buy/install whatever I am building and save time/money.
Please consider seeing things from a different perspective than "your book".
Not saying this to antagonise you, but because a lot of time reading comments like this makes other posters less willing to share their impractical efforts, and would love to read more of those, not less.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#34Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us.
Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#35It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task. And I hope it stays that way, I don't want MCU shortages...
Many artificial environments are useless to tasks they were not designed for until somebody experiments, tests, redesigns, and iterates.
I'll give a specific example apropos of TFA. Computer vision models were never run on MCUs because they were constrained to the point of being useless for this task, but then someone tried the impractical, and now it's trivial[1]
Regarding MCU shortages, you should be worried about the supply chain, but I don't see the impact really being from running LLMs on ESP-32s, of all things.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#36It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task. And I hope it stays that way, I don't want MCU shortages...
How is it useless? If it has an NPU it is literally built to run models.
You're just extremely biased in what you consider to be a useful ML model. For example, for some strange reason you think only LLMs exist. The model must be as big as possible or else it is pointless.
Training custom non-LLM models for specific tasks so they run on a resource constrained device? You must be insane.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#37Earlier quoted context omitted.
Fun and learning is a really good reason. It also reminds me of the damascene: trying to achieve something that doesn't feel possible, and working through all the extreme resource-constrained engineering limits. I think most of my embedded projects aren't that useful, but they've taught me a lot.
I'm not disagreeing with you, I think this goes beyond impractical, it's doomed from the get go. In my book, impractical means "I built a cuckoo wristwatch". Beyond impractical: "I built a cuckoo wristwatch but there was no room for a working mechanism".
Learning how to make a cuckoo wristwatch and everything that involves helps you acquire skills, even if what you build is fundamentally useless.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#38Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us. Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
And this is why I am extremely bullish on Apple.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#39Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us. Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
> tips about dental hygiene
They already can. Philips' higher end models have Bluetooth connections and provide feedback through the app.
> advertise toothpaste
God please no. I don't want to have to look for an adblocker for a toothbrush AI.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#40Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us. Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
PLEASE DRINK A VERIFICATION CAN