i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?
Run a smaller LLM. It won't be as smart but the one shown here isn't either.
Running a 28.9M parameter LLM on an $8 microcontroller
71–79 of 79 posts
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#72Earlier quoted context omitted.
Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD. I think they got something like 10 _seconds per token_ (not tokens per second). EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri
Rig with 6x RTX 5090. I would not say it is RAM only
Unless I am mis-reading the benchmarks, some of those are run as CPU only, no GPU involvement.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#73Earlier quoted context omitted.
where can I learn about those?
Dave "EEVblog" Jones did a review of a (then) $0.03 microcontroller (Padauk) a while back (sorry, I don't know the exact episode). Iirc an important caveat was that it was a one-time programmable (OTP) part. So you buy a bunch of them, programming failure or firmware-under-test doesn't work? -> toss the part. Of course that isn't an issue for a $0.03 part. But it can be an issue in terms of a board you want it on. Ei…
looks like Padauk PMS150C is back up to 0.08$, oh well :(
it's fun reading all the reactions to it (like "it’s cheaper to program a Padauk PMS150C to be a logic-level converter than to just buy a logic level converter"), but the magic is gone more-or-less
but some ideas inside it and padauk's ecosystem as a whole seem impressive https://jaycarlson.net/2019/09/06/whats-up-with-these-3-cent...
edit: PMS152 seems to still be 0.03$ !
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#74Earlier quoted context omitted.
Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD. I think they got something like 10 _seconds per token_ (not tokens per second). EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri
That's incredible. Sure, not practical for most applications, but if you really want a local top tier model, you can run it on anything as long as you are patient. As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, yo…
I used to run Povray as a kid by setting up a scene in the evening and then letting it render all night and admiring the output in the morning.
I could see where you could set up some coding task and let it churn all night.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#75Earlier quoted context omitted.
Meaning, minus possible required bug fixes, all the frontend languages supported on a full instal of either GCC or LLVM.
The wild thing is that this SoC is a heterogenous compute device. It has three different kinds of cores: a pretty beefy arm64 core and two different RISC-V cores: a 1Ghz one for running Linux and a 700Mhz one dedicated to running a real-time operating system. The arm64 core can also run its own OS.
Also a user-programmable 300 MHz 8051 (8 bit) primarily there to manage timers and interrupts and wake up the big 64 bit cores.
Note that the application-processor 1 GHz C906 and Arm A53 are either/or, you can't run both at the same time — similar to the Arm M33 and RISC-V Hazard3 core pairs on the Raspberry Pi RP2350.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#76Earlier quoted context omitted.
> tips about dental hygiene They already can. Philips' higher end models have Bluetooth connections and provide feedback through the app. > advertise toothpaste God please no. I don't want to have to look for an adblocker for a toothbrush AI.
My goto for what I hate about modern tech is toothbrushes having Bluetooth and needing apps. Not that I hate all modern tech but if it needs an app I probably will.
I'm with you there. My fridge has a wifi connection with an app.
The only function it can do is notify me if the door is open. Thats it. Absolute waste of resources. I openend the enclosure and removed the antenna of course.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#77Re: Running a 28.9M parameter LLM on an $8 microcontroller
#78Earlier quoted context omitted.
It really puts into perspective how much of a leech ARM has been on the entire industry. What being a monopoly does to a mfer.
I think that is pretty ungenerous. Before ARM, ISAs were not a commodity, and there were only closed, proprietary implementations of them (usually from a single vendor). Arm licensing its IP and actual designs was hugely beneficial for the broader ecosystem and led to their prevalence in the embedded space. The toolchain and software network effect made it a no-brainer to either reach for a completed Arm design, cont…
They would not have been allowed to, just like Nvidia wasn't. Although actually with the cult like behaviour around Apple that extends to legal stuff who knows really, maybe they would have been allowed to buy.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#79Wont this wear out the flash memory quickly? I wonder hiw many read cycles can it survive
Reads are effectively infinite. I'm not aware of any upper limit shorter than "end of the universe" timescales. Flash wears out from writing and the answer is in the tens to hundreds of thousands of writes per cell.
https://en.wikipedia.org/wiki/Electromigration
See also ds/3ds flash used in game carts. They need to be powered on every so often to avoid bitrot. Their memory controllers use a form of ecc that causes weak bits to be rewritten. But if it's left too long then ecc cannot correct. This is over 15-30 year timescale fwiw.
But all chips will die in like 50-100 years or so because physics gets em with electromigration.