i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?
Run a smaller LLM. It won't be as smart but the one shown here isn't either.
Running a 28.9M parameter LLM on an $8 microcontroller
71–77 of 77 posts
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#72Earlier quoted context omitted.
Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD. I think they got something like 10 _seconds per token_ (not tokens per second). EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri
Rig with 6x RTX 5090. I would not say it is RAM only
Unless I am mis-reading the benchmarks, some of those are run as CPU only, no GPU involvement.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#73Earlier quoted context omitted.
where can I learn about those?
Dave "EEVblog" Jones did a review of a (then) $0.03 microcontroller (Padauk) a while back (sorry, I don't know the exact episode). Iirc an important caveat was that it was a one-time programmable (OTP) part. So you buy a bunch of them, programming failure or firmware-under-test doesn't work? -> toss the part. Of course that isn't an issue for a $0.03 part. But it can be an issue in terms of a board you want it on. Ei…
looks like Padauk PMS150C is back up to 0.08$, oh well :(
it's fun reading all the reactions to it (like "it’s cheaper to program a Padauk PMS150C to be a logic-level converter than to just buy a logic level converter"), but the magic is gone more-or-less
but some ideas inside it and padauk's ecosystem as a whole seem impressive https://jaycarlson.net/2019/09/06/whats-up-with-these-3-cent...
edit: PMS152 seems to still be 0.03$ !
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#74Earlier quoted context omitted.
Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD. I think they got something like 10 _seconds per token_ (not tokens per second). EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri
That's incredible. Sure, not practical for most applications, but if you really want a local top tier model, you can run it on anything as long as you are patient. As someone with a healthy amount of RAM, but just a 16GB GPU, I am wondering what kind of work I could queue up for overnight runs. I thought the best models were fully out of reach, but the 128GB CPU only test had a 1.8 tokens/second. While not speedy, yo…
I used to run Povray as a kid by setting up a scene in the evening and then letting it render all night and admiring the output in the morning.
I could see where you could set up some coding task and let it churn all night.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#75Earlier quoted context omitted.
Meaning, minus possible required bug fixes, all the frontend languages supported on a full instal of either GCC or LLVM.
The wild thing is that this SoC is a heterogenous compute device. It has three different kinds of cores: a pretty beefy arm64 core and two different RISC-V cores: a 1Ghz one for running Linux and a 700Mhz one dedicated to running a real-time operating system. The arm64 core can also run its own OS.
Also a user-programmable 300 MHz 8051 (8 bit) primarily there to manage timers and interrupts and wake up the big 64 bit cores.
Note that the application-processor 1 GHz C906 and Arm A53 are either/or, you can't run both at the same time — similar to the Arm M33 and RISC-V Hazard3 core pairs on the Raspberry Pi RP2350.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#76Earlier quoted context omitted.
> tips about dental hygiene They already can. Philips' higher end models have Bluetooth connections and provide feedback through the app. > advertise toothpaste God please no. I don't want to have to look for an adblocker for a toothbrush AI.
My goto for what I hate about modern tech is toothbrushes having Bluetooth and needing apps. Not that I hate all modern tech but if it needs an app I probably will.
I'm with you there. My fridge has a wifi connection with an app.
The only function it can do is notify me if the door is open. Thats it. Absolute waste of resources. I openend the enclosure and removed the antenna of course.