61–70 of 79 posts
Earlier quoted context omitted.
There are 2 cent microcontrollers. Used in the cheapest devices you wouldn't even imagine to contain any software - eg. To control a touch sensitive on-off switch.
where can I learn about those?
Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us. Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
> tips about dental hygiene They already can. Philips' higher end models have Bluetooth connections and provide feedback through the app. > advertise toothpaste God please no. I don't want to have to look for an adblocker for a toothbrush AI.
Earlier quoted context omitted.
There are 2 cent microcontrollers. Used in the cheapest devices you wouldn't even imagine to contain any software - eg. To control a touch sensitive on-off switch.
where can I learn about those?
Iirc an important caveat was that it was a one-time programmable (OTP) part. So you buy a bunch of them, programming failure or firmware-under-test doesn't work? -> toss the part. Of course that isn't an issue for a $0.03 part. But it can be an issue in terms of a board you want it on. Either that means discarding (breakout) boards too, or for development you'd need some kind of adapter to put bare ICs in.
Such annoyances only make sense for high-volume, low cost applications. Which is eactly where parts like that go in.
While running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weights
In the small parameter regime there is no room for benchmaxxing, so instead of leaderboards becoming useless, their utility was merely reduced to establishing ever smaller models with similar performance on the benchmarks, forcing compression or redundancy to be recognized and eliminated at the modeling level.
It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
It really puts into perspective how much of a leech ARM has been on the entire industry. What being a monopoly does to a mfer.
Also you can reach out to ESP32 as alternative, plenty of maker projects using them.
Earlier quoted context omitted.
Run the numbers before you waste time here. I doubt this will work.
PIO to PIO between two rp2350s should be able to transfer as many bits per clock as you can spare pins for. They have a single cycle double multiply per core, and the interpolators give you a heap of ability The PIO can be awkward, but you can run a bunch of them at once. Going from MCU to MCU you don't even need to involve the CPU cores, PIO to PIO Comms via pins You are obviously not going to get big TOPS from it b…
As someone who has done a reasonable amount with PIO, I do not think this is possible. However, that should not stop you. If you get it to work, please ping me.
Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD. I think they got something like 10 _seconds per token_ (not tokens per second). EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri