Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running a 28.9M parameter LLM on an $8 microcontroller

github.com

Post reply on HN
Print view View original

21–30 of 77 posts

Re: Running a 28.9M parameter LLM on an $8 microcontroller

#21

Post by skippyfish »

NooneAtAll3 wrote: post #19
skippyfish wrote: post #14

Earlier quoted context omitted.

You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task. And I hope it stays that way, I don't want MCU shortages...

isn't floor price 0.10$ for last couple years?

Well, that's sub-$0.50. But yeah, CH32V003 is in that ballpark, and some of the cheapest Microchip and Infineon products are around $0.20.

It's almost never worth it to buy the cheapest chip unless you're making a million of something, but there are very good ones around $1-$2, and $5 is the upscale stuff.

Re: Running a 28.9M parameter LLM on an $8 microcontroller

#22

Post by raphlinus »

Rohansi wrote: post #15
raphlinus wrote: post #12

Earlier quoted context omitted.

If you want to do this at the $1 price point, you can on RP2350, albeit with some limitations. In particular, it maxes out at full speed (12Mbps). The trick is to use the on-chip USB peripheral for one, and connect the other to GPIO pins backed by PIO. This works today with tinyusb and pico-pio-usb, but I'm also playing with a Rust port which I'm hoping will have higher performance.

$8 ish gets you an ESP32-S3 board with PSRAM, flash, and two USB-C ports. The PSRAM and flash are specifically used for this LLM project. I can't find anything like that with the RP2350 for $1.

Entirely fair, $1 is just the chip, not the board. No question the ESP32-S3 is incredibly good value.

Re: Running a 28.9M parameter LLM on an $8 microcontroller

#24

Post by hgoel »

Lerc wrote: post #8
rao-v wrote: post #5

This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!

One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions. PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton o…

Wouldn't a layer per MCU be heavily bandwidth constrained?

Re: Running a 28.9M parameter LLM on an $8 microcontroller

#25

Post by dannyw »

skippyfish wrote: post #14
titzer wrote: post #10

It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!

You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task. And I hope it stays that way, I don't want MCU shortages...

Fun and learning is a really good reason. It also reminds me of the damascene: trying to achieve something that doesn't feel possible, and working through all the extreme resource-constrained engineering limits.

I think most of my embedded projects aren't that useful, but they've taught me a lot.

Re: Running a 28.9M parameter LLM on an $8 microcontroller

#26

Post by monocasa »

hgoel wrote: post #24
Lerc wrote: post #8

Earlier quoted context omitted.

One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions. PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton o…

Wouldn't a layer per MCU be heavily bandwidth constrained?

Depending on where you slice the model up, it can be not a whole lot of data. For instance each transformer block outputs a single vector in an embedding space.

I can see that being cheaper to bitbang with PIO than to actually compute.

There's certainly some latency stack up, but throughput should be remarkably good.

Re: Running a 28.9M parameter LLM on an $8 microcontroller

#27

Post by monocasa »

Rohansi wrote: post #13
chrishynes wrote: post #6

Why can't this scale to run much larger models on CPU backed by flash with good access patterns?

My guess is because the ESP32's flash is only ~1/4 the bandwidth of the internal SRAM. If you do this on a more powerful system not only is the gap much wider but you also have much more compute you need to keep fed with bandwidth to be efficient.

It's also mapped into the address space so there's very little extra latency in grabbing the embedding as opposed to something like nvme that will have to setup a command list, submit it to the drive's microcontroller, wait for the op to be processed, etc.

Re: Running a 28.9M parameter LLM on an $8 microcontroller

#29

Post by Lerc »

NuclearPM wrote: post #20
Lerc wrote: post #8

Earlier quoted context omitted.

One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions. PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton o…

Run the numbers before you waste time here. I doubt this will work.

PIO to PIO between two rp2350s should be able to transfer as many bits per clock as you can spare pins for.

They have a single cycle double multiply per core, and the interpolators give you a heap of ability

The PIO can be awkward, but you can run a bunch of them at once. Going from MCU to MCU you don't even need to involve the CPU cores, PIO to PIO Comms via pins

You are obviously not going to get big TOPS from it because a Trillion is a ridiculous amount anyway. But never underestimate the power of controlling the whole pipeline.

Ultimately none of the other things I'm doing with MCUs are practical, why would this to be any different.

Re: Running a 28.9M parameter LLM on an $8 microcontroller

#30

Post by skippyfish »

dannyw wrote: post #25
skippyfish wrote: post #14

Earlier quoted context omitted.

You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task. And I hope it stays that way, I don't want MCU shortages...

Fun and learning is a really good reason. It also reminds me of the damascene: trying to achieve something that doesn't feel possible, and working through all the extreme resource-constrained engineering limits. I think most of my embedded projects aren't that useful, but they've taught me a lot.

I'm not disagreeing with you, I think this goes beyond impractical, it's doomed from the get go.

In my book, impractical means "I built a cuckoo wristwatch". Beyond impractical: "I built a cuckoo wristwatch but there was no room for a working mechanism".

Post reply on HN