TinyML: Ultra-low power machine learning

TinyML means running machine learning on low power embedded devices, like microcontrollers, with constrained compute and memory. I was supremely lucky in being around for the birth of this stuff: I helped launch TensorFlow Lite for Microcontrollers at Google back in 2019, co-authored the O'Reilly book TinyML (with Pete Warden, who deserves credit more than anyone for making this scene happen) and, ran the initial TinyML meetups at the Google and Qualcomm campuses.

You likely have a TinyML system in your pocket right now: every cellphone has a low power DSP chip running a deep learning model for keyword spotting, so you can say "Hey Google" or "Hey Siri" and have it wake up on-demand without draining your battery. It’s an increasingly pervasive technology.

TinyML is a subset of edge AI, which includes any type of device sitting at the edge of a network. This has grown far beyond the general purpose microcontrollers we were hacking on in the early days: there are now a ton of highly capable devices designed specifically for low power deep learning inference.

It’s astonishing what is possible today: real time computer vision on microcontrollers, on-device speech transcription, denoising and upscaling of digital signals. Generative AI is happening, too, assuming you can find a way to squeeze your models down to size. We are an unsexy field compared to our hype-fueled neighbors, but the entire world is already filling up with this stuff and it’s only the very beginning. Edge AI is being rapidly deployed in a ton of fields: medical sensing, wearables, manufacturing, supply chain, health and safety, wildlife conservation, sports, energy, built environment—we see new applications every day.

This is an unbelievably fascinating area: it’s truly end-to-end, covering an entire landscape from processor design to deep learning architectures, training, and hardware product development. There are a ton of unsolved problems in academic research, practical engineering, and the design of products that make use of these capabilities.

I’ve worked in many different parts of tech industry and this one feels closest to capturing the feeling I’ve read about in books about the early days of hacking with personal computers. It’s fast growing, tons of really hard problems to solve, even more low hanging fruit, and has applications in almost every space.

If you’re interested in getting involved, you can choose your own adventure: learn the basics and start building products, or dive deep and get involved with research. Here are some resources:

* Harvard TinyML course: https://www.edx.org/learn/machine-learning/harvard-universit...

* Coursera intro to embedded ML: https://www.coursera.org/learn/introduction-to-embedded-mach...

* TinyML (my original book, on the absolute basics. getting a bit out of date, contact me if you wanna help update it): https://tinymlbook.com

* AI at the Edge (my second book, focused on workflows for building real products): https://www.amazon.com/AI-Edge-Real-World-Problems-Embedded/...

* ML systems with TinyML (wiki book by my friend Prof. Vijay Reddi at Harvard): https://harvard-edge.github.io/cs249r_book/

* TinyML conference: https://www.tinyml.org/event/summit-2024/

* I also write a newsletter about this stuff, and the implications it has for human computer interaction: https://dansitu.substack.com

I left Google 4 years ago to lead the ML team at Edge Impulse (http://edgeimpulse.com) — we have a whole platform that makes it easy to develop products with edge AI. Drop me an email if you are building a product or looking for work: daniel@edgeimpulse.com

For inference there's a whole spectrum of approaches that let you can trade off flexibility for performance. TF Lite Micro is at one end, hand-written Verilog is at the other.

Typically, flexibility is more important at the start of a project, while deep optimization is more important later. You wanna be able to iterate fast. That said, the flexible approaches are now good enough that you will typically get better ROI from optimizing your model architecture rather than your inference code.

I think the sweet spot today is code-generation, when targeting general purpose cores. There's also increasing numbers of chips with hardware acceleration, which is accessed using a compiler that takes a model architecture as input.