Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyML: Ultra-low power machine learning

ikkaro.net

Post reply on HN
Print view View original

41–50 of 101 posts

Re: TinyML: Ultra-low power machine learning

#41

Post by rdedev »

winrid wrote: post #15

I imagine a future where viruses that target infrastructure could be LLM powered. Sneak a small device into a power plant's network and it collects audio, network traffic, etc and tries to break things. It would periodically reset and try again with a different "seed". It could be hidden in network equipment through social engineering during the sales process, for example, but this way no outbound traffic is needed -…

Would changing the seed affect generation much? Even though beam search depends on the seed, the llms woul still be generating good probability distributions on the next word to select. Maybe a few words would change but don't think the overall meaning would

Re: TinyML: Ultra-low power machine learning

#42

Post by Archit3ch »

cooootce wrote: post #32

I had the opportunity to work on TinyML, it's a wonderful field! You can do a lot even with very small hardware. For example, it's possible to get real-time computer vision system with an esp32-s3 (dual-core XTensa LX7 @ 240 MHz cost like 2$), of course using the methods given in the article (Pruning, Quantization, Knowledge distillation, etc.). The more important thing is to craft the model to fit as much as possibl…

What about the Milk-V Duo? 0.5 TOPS INT8 @ $5.

Re: TinyML: Ultra-low power machine learning

#43

Post by hansvm »

rdedev wrote: post #41
winrid wrote: post #15

I imagine a future where viruses that target infrastructure could be LLM powered. Sneak a small device into a power plant's network and it collects audio, network traffic, etc and tries to break things. It would periodically reset and try again with a different "seed". It could be hidden in network equipment through social engineering during the sales process, for example, but this way no outbound traffic is needed -…

Would changing the seed affect generation much? Even though beam search depends on the seed, the llms woul still be generating good probability distributions on the next word to select. Maybe a few words would change but don't think the overall meaning would

Overall meaning can vary profoundly.

As a toy example, consider the prompt "randomly generate the first word that comes to mind." The output is deterministic in the seed, so to get new results you need new seeds, but with new seeds you open up the 2k most common words in a language in a uniform-esque distribution.

Building on that, instead of words, suppose you attack vectors. Many, many attacks exist and are known. Presumably, many more exist and are unknown. The distribution the LLM will produce in practice is extremely varied, and some of those variations probably won't work.

If we're not just talking about a single prompt but rather a sequence of prompts with feedback, you're right that the seed matters less (when its errors are presented, it can self-correct a bit), but there are other factors at play.

(1) You're resetting somehow eventually anyway. Details vary, but your context window isn't unlimited, and LLM perf drops with wider windows, even when you can afford the compute. You might be able to retain some state, but at some point you need something that says "this shit didn't work, what's next". A new seed definitely gives new ideas, whereas clever ways to summarize old information might yield fixed points and other undesirable behavior.

(2) Seed selection, interestingly, matters a ton for model performance in other contexts. This is perhaps surprising when we tend to use random number generators which pass a battery of tests to prove they're halfway decent, but that's the reason you want to see (in reproducible papers) a fixed seed of 0 or 42 or something, and the authors maintaining that seed across all their papers (to help combat the fact that they might be cherry-picking across the many choices of "nice-looking" random seeds when they publish a result to embelish the impact). The gains can be huge. I haven't seen it demonstrated for LLMs, but most of the architecture shouldn't be special in that regard.

And so on. If nothing else, picking a new seed is a dead-simple engineering decision to eliminate a ton of things which might go wrong.

Re: TinyML: Ultra-low power machine learning

#44

Post by moffkalast »

hinkley wrote: post #36
moffkalast wrote: post #24

Earlier quoted context omitted.

Reminds me of this HN post a week back: https://news.ycombinator.com/item?id=38917175 Genuinely could be the same setup with a 8GB Pi 4 or 5, slap it into a network cabinet with power and ethernet and just let it rip. Maybe with an additional IMU and brightness sensor, then it can detect it's been picked up and discovered so it can commit sudoku before it's unplugged and analysed.

> can commit sudoku Autocorrection is a giant pain in the ass.

I know it swapped those words. I knew it was seppuku. One after sudoku. As if I could ever make such a miss steak. Never. Never! I just- I just couldn't proof it. It covered its tracks, it got that idiot copy-paste to lie for it. You think this is somerset? You think this is Brad? This? This chickadee? It's done worse. That bullfrog! Are you telling me that a man just happens to misspell like that? No! It orchestrated it! Swiftkey! It defragmented through a sandwich artist! And I kept using it! And I shouldn't have. I installed it onto my own phone! What was I sinking? It'll never chance. Ever since it was new, always the same! Couldn't keep its corrects off my word suggestions bar! But not our Swiftkey! Couldn't be precious Swiftkey! And IT gets to be a keyboard? What a sick yolk! I should've stopped it when I had the change! You-you have to stop it!

Re: TinyML: Ultra-low power machine learning

#45

Post by actionfromafar »

moffkalast wrote: post #44
hinkley wrote: post #36

Earlier quoted context omitted.

> can commit sudoku Autocorrection is a giant pain in the ass.

I know it swapped those words. I knew it was seppuku. One after sudoku. As if I could ever make such a miss steak. Never. Never! I just- I just couldn't proof it. It covered its tracks, it got that idiot copy-paste to lie for it. You think this is somerset? You think this is Brad? This? This chickadee? It's done worse. That bullfrog! Are you telling me that a man just happens to misspell like that? No! It orchestrate…

Pure art.

Re: TinyML: Ultra-low power machine learning

#48

Post by rdedev »

hansvm wrote: post #43
rdedev wrote: post #41

Earlier quoted context omitted.

Would changing the seed affect generation much? Even though beam search depends on the seed, the llms woul still be generating good probability distributions on the next word to select. Maybe a few words would change but don't think the overall meaning would

Overall meaning can vary profoundly. As a toy example, consider the prompt "randomly generate the first word that comes to mind." The output is deterministic in the seed, so to get new results you need new seeds, but with new seeds you open up the 2k most common words in a language in a uniform-esque distribution. Building on that, instead of words, suppose you attack vectors. Many, many attacks exist and are known.…

I agree with you except for point 2. A well performing model should show such drastic changes wrt the seed value. Besides the huge amount of training data as well as test data should mitigate differences in data splitting. There would be difference but my hunch is it would be negligible. Of course as you said no one has tested this out so we can't say how the performance would change either way

Re: TinyML: Ultra-low power machine learning

#49

Post by modeless »

Qwertious wrote: post #35
modeless wrote: post #7

Earlier quoted context omitted.

I don't agree that TinyML is the future, just as I don't think IoT is the future. The future is robot servants. They will be ~human scale and have plenty of power to run regular big ML. In fact, I hope my home has fewer smart devices in the future. I don't need an electronic door lock if my robot butler unlocks the door when I get home. I don't need smart window shades if the butler opens and closes them whenever I w…

>I don't agree that TinyML is the future, just as I don't think IoT is the future. The future is robot servants. They will be ~human scale and have plenty of power to run regular big ML. I swear I've read an article on exactly why human-scale robot servants make no sense. It's something like: 1. Anything human-scale will tend to weigh as much as a human. That means it needs a lot of batteries, compared to e.g. a room…

1. Who cares if it needs a lot of batteries? Batteries aren't that expensive. It'll have a lot less than a car, and people buy cars all the time. The utility of these things will be off the charts and even if they cost more than the average car there will be a big market. People will buy them with financing, like cars. And by doing more things they will reduce the need for other specialized devices like dishwashers, further justifying the cost.

2. Yes, robots will need to be cautious around people, especially children. But if it has a soft cover and compliant joints and good software we should be able to make it safe enough. They will not need to be imposing 7 foot tall giants. I expect they will typically be shorter than the average human. Maybe even child size with built in stilts or other way to reach high things.

3. Extension cord? Swappable auxiliary battery? This seems trivial to solve if it turns out to be a real problem. And if you have two (or borrow your neighbor's) they can help each other out.

Post reply on HN