Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyML: Ultra-low power machine learning

ikkaro.net

Post reply on HN
Print view View original

71–80 of 101 posts

Re: TinyML: Ultra-low power machine learning

#71

Post by Cacti »

cooootce wrote: post #32

I had the opportunity to work on TinyML, it's a wonderful field! You can do a lot even with very small hardware. For example, it's possible to get real-time computer vision system with an esp32-s3 (dual-core XTensa LX7 @ 240 MHz cost like 2$), of course using the methods given in the article (Pruning, Quantization, Knowledge distillation, etc.). The more important thing is to craft the model to fit as much as possibl…

thank you for the post and good work.

can I ask, is the focus primarily on inference? is there anything serious going on with training at the power scale you are talking about?

Re: TinyML: Ultra-low power machine learning

#72

Post by Reviving1514 »

anigbrowl wrote: post #69
cooootce wrote: post #32

I had the opportunity to work on TinyML, it's a wonderful field! You can do a lot even with very small hardware. For example, it's possible to get real-time computer vision system with an esp32-s3 (dual-core XTensa LX7 @ 240 MHz cost like 2$), of course using the methods given in the article (Pruning, Quantization, Knowledge distillation, etc.). The more important thing is to craft the model to fit as much as possibl…

Great post. surprised and excited to discover Tensorflow models can run on commodity hardware like the ESP32.

I ended up hand rolling a custom micropython module for the S3 to do a proof of concept handwriting detection demo on an ESP32, might be interesting to some.

https://luvsheth.com/p/running-a-pytorch-machine-learning

Re: TinyML: Ultra-low power machine learning

#73

Post by Cacti »

anigbrowl wrote: post #69
cooootce wrote: post #32

I had the opportunity to work on TinyML, it's a wonderful field! You can do a lot even with very small hardware. For example, it's possible to get real-time computer vision system with an esp32-s3 (dual-core XTensa LX7 @ 240 MHz cost like 2$), of course using the methods given in the article (Pruning, Quantization, Knowledge distillation, etc.). The more important thing is to craft the model to fit as much as possibl…

Great post. surprised and excited to discover Tensorflow models can run on commodity hardware like the ESP32.

Problems reducible even partially to matrix math are for many practical purposes embarrassing parallel even within a single core. A couple hundred million FLOPS with 1990s SIMD support will let you run nearly all near-SOTA models within, idk, 3s, with most running in 0.1 or 0.01s. That’s pretty fast considering it’s an EP32 and some of these capabilities/models didn’t even exist a year ago.

Your expectation was not really wrong, because for most purposes, when discussing a “model” one is really talking about “capabilities”. And capabilities often require many calls to the model. And that capability may be reliant on being refreshed very rapidly… and now your 0.1s is not even slow, it’s almost existentially slow.

Re: training. even on the EP32, training is entirely doable, so long as you pretend you are in 2011 solving 2011 problems hahaha

Re: TinyML: Ultra-low power machine learning

#74

Post by two_in_one »

Qwertious wrote: post #35
modeless wrote: post #7

Earlier quoted context omitted.

I don't agree that TinyML is the future, just as I don't think IoT is the future. The future is robot servants. They will be ~human scale and have plenty of power to run regular big ML. In fact, I hope my home has fewer smart devices in the future. I don't need an electronic door lock if my robot butler unlocks the door when I get home. I don't need smart window shades if the butler opens and closes them whenever I w…

>I don't agree that TinyML is the future, just as I don't think IoT is the future. The future is robot servants. They will be ~human scale and have plenty of power to run regular big ML. I swear I've read an article on exactly why human-scale robot servants make no sense. It's something like: 1. Anything human-scale will tend to weigh as much as a human. That means it needs a lot of batteries, compared to e.g. a room…

Just asked the latest gpt preview model to explain why human sized robots make no sense, then why they are the future. In both cases it managed to provide 10 arguments. Some of them are similar, like 'social acceptance' in negative part and 'Sociocultural Acceptance' in positive.

PS: it doesn't accept $500 tips anymore ;)

Re: TinyML: Ultra-low power machine learning

#75

Post by DoingIsLearning »

wegfawefgawefg wrote: post #68
flyingcircus3 wrote: post #67

Earlier quoted context omitted.

Why not three models? One model does basic feature detections, like lines, shapes, etc. A second model that can take the first model's output as its input, and identify birds. A third model can take the first model's output as its input, and identify houses.

This is a lesson I've watched people, and companies learn for the past 7-8 years. An end to end model will always outperform a sequence of models designed to target specific features. You truncate information when you render the data into output space (the model output vector) from feature space (much richer data inside the model), thats the primary reason why to do transfer learning all layers are frozen, the final…

As someone not in ML but curious about the field this is really interesting. Intuitively indeed it would be natural to aim for some sort of inspectable composition of models.

Is there specific tooling to inspect intermediate layers or will they be unintelligible for humans?

Re: TinyML: Ultra-low power machine learning

#76

Post by flyingcircus3 »

wegfawefgawefg wrote: post #68
flyingcircus3 wrote: post #67

Earlier quoted context omitted.

Why not three models? One model does basic feature detections, like lines, shapes, etc. A second model that can take the first model's output as its input, and identify birds. A third model can take the first model's output as its input, and identify houses.

This is a lesson I've watched people, and companies learn for the past 7-8 years. An end to end model will always outperform a sequence of models designed to target specific features. You truncate information when you render the data into output space (the model output vector) from feature space (much richer data inside the model), thats the primary reason why to do transfer learning all layers are frozen, the final…

I'm genuinely confused at how you made these assumptions about what I'm describing. Because the "more correct" design you contrast with the strawman you've concluded I'm describing is actually what I'm talking about, if perhaps imprecisely. A pretrained model like mobilenetV2, with its final layer removed, and custom models trained on bird and house images, which take this mobilenetv2[:-1] output as input. MobilenetV2 is 2ish megabytes at 224x224, and these final bird and house layers will be kilobytes. Having two multiple-megabyte models that are 95% identical is a giant waste of our embedded target's resources. It also means that a scheme that processed a single image with two full models (instead of one big, two small) would spend 95% of the second full model's processing time redundantly performing the same operations on the same data. Breaking up the models across two stages produces substantial savings of both processing time and flash storage, with a single big model as the "feature detection" first stage of both overall inferences, with small specialized models as a second stage.

Re: TinyML: Ultra-low power machine learning

#77

Post by espe »

winrid wrote: post #15

I imagine a future where viruses that target infrastructure could be LLM powered. Sneak a small device into a power plant's network and it collects audio, network traffic, etc and tries to break things. It would periodically reset and try again with a different "seed". It could be hidden in network equipment through social engineering during the sales process, for example, but this way no outbound traffic is needed -…

and the other way round - have it built in for self-fuzzing and healing the infra.

Re: TinyML: Ultra-low power machine learning

#80

Post by wegfawefgawefg »

DoingIsLearning wrote: post #75
wegfawefgawefg wrote: post #68

Earlier quoted context omitted.

This is a lesson I've watched people, and companies learn for the past 7-8 years. An end to end model will always outperform a sequence of models designed to target specific features. You truncate information when you render the data into output space (the model output vector) from feature space (much richer data inside the model), thats the primary reason why to do transfer learning all layers are frozen, the final…

As someone not in ML but curious about the field this is really interesting. Intuitively indeed it would be natural to aim for some sort of inspectable composition of models. Is there specific tooling to inspect intermediate layers or will they be unintelligible for humans?

The unending quest for "Explainability" has yielded some tools but has been utterly overrun and outpaced by newer more complicated architectures and unfathomably large models. (Banks and insurance, finance etc really want explainability for auditing.)

The early layers in a vision model are sort of interpetable. They look like lines and dots and scratchy patterns being composited. You can see the exact same features in L1 and L2 biological neural networks in cats, monkeys, mice, etc. As you get deeper into the network the patterns become really abstract. For a human, the best you can do is render a pattern of inputs that maximizes a target internal neurons activation to see what it detects.

You can sort of see what they represent in vision. Dogs, fur, signs, face, happy, sad, etc, but once its a multimodal model and there is time and language involved it gets really difficult. And at that point you might as well just use the damn thing, or just ask it.

In finance, you cant tell what the fuck any of the feature detectors are. Its just very abstract.

As for tooling, a little bit of numpy and pytorch, dump some neurpn weights to a png, there you go. Download a small convnet pretrained network, amd i bet gpt4 can walk you through the process.

Post reply on HN