Wont this wear out the flash memory quickly? I wonder hiw many read cycles can it survive
Running a 28.9M parameter LLM on an $8 microcontroller
51–60 of 79 posts
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#52Wont this wear out the flash memory quickly? I wonder hiw many read cycles can it survive
https://docs.espressif.com/projects/esp-idf/en/stable/esp32/...
It's write and/or erase cycles that cause wear.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#53Earlier quoted context omitted.
There's a neat project out there that can read your water meter into home assistant using an ESP32+camera+computer vision. I imagine a small TPU like this could be very useful for similar projects. More compute means higher resolutions and more reliability.
Interested in this. Link please
AI on the edge https://jomjol.github.io/AI-on-the-edge-device-docs/
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#54How accurate is this quantized model?
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#55Re: Running a 28.9M parameter LLM on an $8 microcontroller
#56Wont this wear out the flash memory quickly? I wonder hiw many read cycles can it survive
Reads are effectively infinite. I'm not aware of any upper limit shorter than "end of the universe" timescales.
Flash wears out from writing and the answer is in the tens to hundreds of thousands of writes per cell.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#57Earlier quoted context omitted.
Don't forget the 128 bit vector ISA with 32 registers, supporting up to 64 bit int and FP, and with LMUL=8 you can process 1024 bits with a single instruction (at 3 cycles per 128 bits for most operations). Fully supported by GCC and CLANG (xTHeadVector) and compatible with RVV 1.0 with just a command line switch if you use the C intrinsic functions. (a lot of code working on 8 bit elements is binary compatible with…
Meaning, minus possible required bug fixes, all the frontend languages supported on a full instal of either GCC or LLVM.
The wild thing is that this SoC is a heterogenous compute device. It has three different kinds of cores: a pretty beefy arm64 core and two different RISC-V cores: a 1Ghz one for running Linux and a 700Mhz one dedicated to running a real-time operating system. The arm64 core can also run its own OS.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#58It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards: https://milkv.io The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
It really puts into perspective how much of a leech ARM has been on the entire industry. What being a monopoly does to a mfer.
I think that is pretty ungenerous. Before ARM, ISAs were not a commodity, and there were only closed, proprietary implementations of them (usually from a single vendor). Arm licensing its IP and actual designs was hugely beneficial for the broader ecosystem and led to their prevalence in the embedded space. The toolchain and software network effect made it a no-brainer to either reach for a completed Arm design, contract a customized one, or build your own.
The arm experiment ran its course though; the power one vendor had in the marketplace started to be abused for the benefit of the IP holder and detriment to others. Now RISC-V is going one step further with a completely open ISA and also completely open designs. This is an excellent development in the nick of time.
I wonder how Apple feels about arm64 and RISC-V now. They could have probably bought ARM at any point but maybe never considered it to avoid anti-monopoly blowback.
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#59i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?
Re: Running a 28.9M parameter LLM on an $8 microcontroller
#60Earlier quoted context omitted.
Meaning, minus possible required bug fixes, all the frontend languages supported on a full instal of either GCC or LLVM.
The wild thing is that this SoC is a heterogenous compute device. It has three different kinds of cores: a pretty beefy arm64 core and two different RISC-V cores: a 1Ghz one for running Linux and a 700Mhz one dedicated to running a real-time operating system. The arm64 core can also run its own OS.
I haven't noticed it, rather cool!