11–20 of 79 posts
Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
I think they got something like 10 _seconds per token_ (not tokens per second).
EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri
>esp32-s3 This microcontroller is a beast, currently using it to do dev work on a pi4. Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
This works today with tinyusb and pico-pio-usb, but I'm also playing with a Rust port which I'm hoping will have higher performance.
Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
And I hope it stays that way, I don't want MCU shortages...
>esp32-s3 This microcontroller is a beast, currently using it to do dev work on a pi4. Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
If you want to do this at the $1 price point, you can on RP2350, albeit with some limitations. In particular, it maxes out at full speed (12Mbps). The trick is to use the on-chip USB peripheral for one, and connect the other to GPIO pins backed by PIO. This works today with tinyusb and pico-pio-usb, but I'm also playing with a Rust port which I'm hoping will have higher performance.
It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task. And I hope it stays that way, I don't want MCU shortages...
Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD. I think they got something like 10 _seconds per token_ (not tokens per second). EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri
As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, you could probably do something with that given extensive coffee breaks. This speed simulator[0] demos what it looks like.
It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
When I bought my 64 MB Duo they were $3!
Then for a long time they were $5 for the 64 MB, $7 for the 256 MB, and $10 for the 512 MB.
Sadly, like everything else, they've gone up considerably this year.
It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task. And I hope it stays that way, I don't want MCU shortages...
This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions. PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton o…