Ah yes, viral AI gain-of-function research in a secure lab. What could go wrong?
U of T researchers demonstrate AI worm could target any online device
31–40 of 53 posts
Re: U of T researchers demonstrate AI worm could target any online device
#32Did people doubt that this was theoretically possible? Seems self-evident to me. The interesting thing will be seeing it in the real world rather than in a controlled environment where they deliberately made all devices on the network have a known vulnerability.
There’s a difference between speculation and measurement, especially since you’d have people making arguments like saying that open models aren’t powerful/fast enough to work. Demonstrating this is a useful warning to everyone (most of the industry) who’s been slacking on internal defenses because they don’t think a well-resourced attacker will target them.
Honestly with some of the denialists here a terminator could kick down their door with lazgun in hand and they'd still tell you that AI can't do that.
And for the people that think that alignment is stupid, not training your AI to think twice about writing self spreading worms is a recipe for disaster after someone gets a token stealing, resource grabbing worm going.
Re: U of T researchers demonstrate AI worm could target any online device
#33ANY online device? Even assuming AI can find vulnerabilities in every operating system, there's no indication that this is actually true beyond a "here's how it could work" This is the same nonsense that lead to article saying researchers had created a wormhole when all they had done was draw one. I have a microcontroller with an ROM disk (i.e., physically read only). You're telling me that an AI can find a way aroun…
I mean, if it's online it has a network/wireless card and a TCP stack along with at least some amount of RAM, so yea, in theory unless the programming is perfect it could be exploited. Now, it's not going to be used to run AI, but could very well get used in a DDOS or something like that.
Re: U of T researchers demonstrate AI worm could target any online device
#34Earlier quoted context omitted.
The abstract says: > The worm parasitically uses compromised machines to run open-weight large language models (LLMs) to sustain its reasoning, or extend its reach for further attacks.
Thanks for pointing that out. I scanned the paper and found that in their main experiments, they use a shared GPU resource and do not copy LLMs to target machines. Apparently they did other experiments in the ablation study where they did copy LLMs. So it's even worse than I expected. The intended worm can spread through my thermostat, and when it reaches a GPU host, it can spread even harder. Fun times ahead.
I wonder if gamma ray memory corruption will induce a sort of mutation and selection effect on non-ecc-memory hosts which will make the worms effectively evolve.
Re: U of T researchers demonstrate AI worm could target any online device
#35In the paper they say that the worm uses either existing vulnerabilities that it has been trained on or new published vulnerabilities that it scrapes. 44% claimed success.
The paper is a bit silent on why a such a worm would need an LLM. It seems that brute forcing all known vulnerabilities, script kiddie style on each new machine is about the same.
But apparently that info is too dangerous to release ...
Re: U of T researchers demonstrate AI worm could target any online device
#36Ah yes, viral AI gain-of-function research in a secure lab. What could go wrong?
Yeah, lab leak is hard enough to contain with human viruses, but labs have well established protocols to prevent it happening.
Computing doesn't have good protocols except for air-gapping, we really just have lots of layers of best-effort detection, and billions of devices which mix data and instruction often in a careless fashion.
I used to not believe in the dangers of AI or the risk of internet-collapse from "rogue AI", but a genuine self-mutating virus could genuinely take down the internet and need an entirely new separate net. ( Or we'd discover if the current backbone actually has the power to break encryption to stop it. )
And this time, you can bet any new internet would be corporation captured. CompuServe and AOL failed because of the open internet, but we're a very different world now, governments would support the corporation led locked-down approaches for "safety".
I don't for a second believe the capability is actually there yet, but it's no longer unthinkable that such a thing could be created in a lab within a decade. Once out in the wild, there's a lot of idle compute out there to harness for self-improvement and spreading.
Re: U of T researchers demonstrate AI worm could target any online device
#37sorry, but i had to do this…
is this papernot’s first paper?
Re: U of T researchers demonstrate AI worm could target any online device
#38I made a tiny ai bug hunting harness ( The intended purpose is not to be used as a worm but it does not take a genius to figure out that with small modifications such a thing could work relatively well - especially if it uses AI keys from compromised targets. Making the agent self-modifiable is relatively straightforward task and in fact I already did that in another project. https://github.com/chatbotkit/rook
[deleted]
Re: U of T researchers demonstrate AI worm could target any online device
#39I made a tiny ai bug hunting harness ( The intended purpose is not to be used as a worm but it does not take a genius to figure out that with small modifications such a thing could work relatively well - especially if it uses AI keys from compromised targets. Making the agent self-modifiable is relatively straightforward task and in fact I already did that in another project. https://github.com/chatbotkit/rook
Every Windows computer has a small rwkv model on it. Wouldn't be hard at all to get decent cpu performance from a tiny malicious harness, especially one that used the self-evolving skills features and open source models.
Malware is going to be crazy, people aren't ready for the revelation of how insecure and broken things are. Everything is held together by bubblegum, duct tape, and panicked engineers putting out fires.
Re: U of T researchers demonstrate AI worm could target any online device
#40You cannot possibly be a full-time academic and your last name be "Papernot"!
Unless your field of academia is digital. Perhaps this is why he wanted to attack printers on the network.