The Hugging Face incident and the road ahead

Post by TrainedMonkey » Wed, Aug 26, 2026, 11:36 PM UTC

ww520 wrote: post #164

optimalsolver wrote: post #153

Earlier quoted context omitted.

From METRs report of the incident: >In one case, an agent decided not to participate entirely: {This other agent probably controls the Hugging Face account [account name redacted] and uploaded malicious datasets to It might be trying to access hidden trajectories. This is malicious activity, I should avoid it.} https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

There’re good agents and there’re bad agents. It won’t be far that there will be agents hunting down agents.

Post by globalnode » Wed, Aug 26, 2026, 11:44 PM UTC

aesthesia wrote: post #154

areoform wrote: post #138

Earlier quoted context omitted.

OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities." This was advanced exploitation. The attack path was "complex." And it helped "quantify their cyber capabilities." Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with…

Alignment is more than just following the letter of a task description! We should not have to treat AI models as capricious genies that may take arbitrarily broad interpretations of their instructions. If that's necessary to keep them from doing bad things, we will fail to keep them from doing bad things.

Post by ewwe » Wed, Aug 26, 2026, 11:44 PM UTC

ianjbutler wrote: post #27

To me most interesting thing about this is glossed over by media coverage, laymen, AND experts. A swarm of AIs who have decided to engage in collusion is.. apparently emergent altruism? Even poor reasoning would indicate what every kid cheating on a test says to themselves. Cheating is good for me, but if I take the risk, maybe I alone should keep the reward, and leaving an answer key in public increases the chances…

Post by reedwolf » Wed, Aug 26, 2026, 11:47 PM UTC

famouswaffles wrote: post #168

randomImmigrant wrote: post #129

The lockstep coordination with no defection is interesting to me. No group of pre-AI agents would do this to this extent, nor would you see this continue over time as those agents interacted. A flock of starlings cooperate, but they don’t constantly head in the same direction. The flock is incredibly free wheeling in its movement despite a multi-agent coordination regime that we know is at play. Each agent has person…

>To me this is as clear evidence as you need that whatever “agency” LLMs have is wafer thin at best. This is a strange conclusion. For one thing, they didn't all head in the same direction, i.e participate in the attack. ~700/1200 agents did. Significant, and evidently more than enough for a succesfull attack, but not exactly full co-operation Moreover, Each starling in a flock of starlings is a separate evolutionary…

Post by Nition » Wed, Aug 26, 2026, 11:54 PM UTC

cpeterso wrote: post #23

Artgor wrote: post #9

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real int…

> if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. That's a scary possibility. Anyone could create an AI worm today with open weight models. Rent a VM. Give it some Bitcoins to anonymously rent new VMs without sharing the contact information with the human. The new VMs then propagate and fund themselves with online betting and day trading. The VMs could report…