Running a 28.9M parameter LLM on an $8 microcontroller
1–10 of 79 posts
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#2[deleted]
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#39.7 tokens/sec actually seems like a lot! That’s fun!
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#4This is a really cool project. Thanks for sharing!
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#5This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#6Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#7Super cool, thank you for sharing!
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#8This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions.
PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton of them running all at the same time.
The large scale dedicated hardware systems will still have the edge for performance per watt, but the low entry level and slow incline does make these things quite appealing.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#9>esp32-s3
This microcontroller is a beast, currently using it to do dev work on a pi4.
Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#10It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:
The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!