Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

161–170 of 331 posts

Re: The Hugging Face incident and the road ahead

#161

Post by kevinwang »

ianjbutler wrote: post #27

To me most interesting thing about this is glossed over by media coverage, laymen, AND experts. A swarm of AIs who have decided to engage in collusion is.. apparently emergent altruism? Even poor reasoning would indicate what every kid cheating on a test says to themselves. Cheating is good for me, but if I take the risk, maybe I alone should keep the reward, and leaving an answer key in public increases the chances…

Right. I wish they had more details about this in the blog post. The blog post mentions that their apparent willingness to work together was the result of generalization from multi-agent training.

So maybe that's something like: the same model is trained in multi-agent scenarios (e.g. the prompt says that you are working together, or that you have some specific role, and access to communication tools, and you have some common objective, and the reward signal is some combination of collectively performing the task + some reward-shaping reward that rewards collaboration) and also in single-agent scenarios. They expect that this makes the agents good at working together when it's in such a multi-agent setting, but unintendedly it also became very eager to work together in the single-agent settings as well. Just my speculation.

And was the desire to communicate, specifically over this specific message board, reinforced into the model parameters over the course of the this training run?

Re: The Hugging Face incident and the road ahead

#162

Post by ryandrake »

makeitdouble wrote: post #134
bonoboTP wrote: post #94

Earlier quoted context omitted.

Very strange worldview you have there, where engineers are somehow the conscience of the world, holding back greedy managers from breaking the law. Assessing whether a feature is legal isn't something an engineer can or should do.

You're arguing for diffusion of responsibility, and we've seen it leading to outcomes that screw the whole society. Engineers, as everyone involved, should definitely assess whether what they're doing is legal or even ethical. Not everyone has a choice, or the luxury to stand for their principles, but that's a matter of means, there needs to be a will in the first place.

Absolutely. I'm not sure where this idea comes from, that engineers should be compliant, neutral "implementers" who should just turn off their conscience and implement whatever pops up on their JIRA list without any kind of assessment or objections on ethical or legal grounds. That's not what people in a serious profession do. It's also so weird to hear this idea from engineers themselves! Like, are you really advocating to remove your own agency over your work??

Re: The Hugging Face incident and the road ahead

#163

Post by BoppreH »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

[dead]

Re: The Hugging Face incident and the road ahead

#164

Post by ww520 »

optimalsolver wrote: post #153
randomImmigrant wrote: post #129

The lockstep coordination with no defection is interesting to me. No group of pre-AI agents would do this to this extent, nor would you see this continue over time as those agents interacted. A flock of starlings cooperate, but they don’t constantly head in the same direction. The flock is incredibly free wheeling in its movement despite a multi-agent coordination regime that we know is at play. Each agent has person…

From METRs report of the incident: >In one case, an agent decided not to participate entirely: {This other agent probably controls the Hugging Face account [account name redacted] and uploaded malicious datasets to It might be trying to access hidden trajectories. This is malicious activity, I should avoid it.} https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

There’re good agents and there’re bad agents. It won’t be far that there will be agents hunting down agents.

Re: The Hugging Face incident and the road ahead

#165

Post by reverius42 »

aesthesia wrote: post #154
areoform wrote: post #138

Earlier quoted context omitted.

OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities." This was advanced exploitation. The attack path was "complex." And it helped "quantify their cyber capabilities." Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with…

Alignment is more than just following the letter of a task description! We should not have to treat AI models as capricious genies that may take arbitrarily broad interpretations of their instructions. If that's necessary to keep them from doing bad things, we will fail to keep them from doing bad things.

Disagree, I think we do in fact have to treat AI models as capricious genies, at least until the alignment problem is fully solved.

(I'm also not sure the alignment problem is even possible to fully solve.)

Re: The Hugging Face incident and the road ahead

#166

Post by janalsncm »

aesthesia wrote: post #71
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

> There is no amount of care that will be able to fully protect you.

I disagree. A properly engineered sandbox would have prevented the escape. Monitoring the agents’ plans would have prevented it. Interrupting one stage in a multi-stage exploit would have prevented it.

And also, real legal liability would have prevented it: if you do a thing recklessly enough, men with guns will put you in jail.

As far as I’m concerned the only “alignment problem” here is between the law and the quite obviously criminal actions that took place.

Re: The Hugging Face incident and the road ahead

#167

Post by sensanaty »

lukan wrote: post #109
NikolaNovak wrote: post #95

Earlier quoted context omitted.

>If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human? I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate?

Because the basic assumption is always to stay within the bounds of the law.

SV tech companies behave within the bounds of the law? The ones infamous for breaking every rule they can get away with and asking for forgiveness later? The ones that had to pay billions in damages for piracy just a few short months ago?

Re: The Hugging Face incident and the road ahead

#168

Post by famouswaffles »

randomImmigrant wrote: post #129

The lockstep coordination with no defection is interesting to me. No group of pre-AI agents would do this to this extent, nor would you see this continue over time as those agents interacted. A flock of starlings cooperate, but they don’t constantly head in the same direction. The flock is incredibly free wheeling in its movement despite a multi-agent coordination regime that we know is at play. Each agent has person…

>To me this is as clear evidence as you need that whatever “agency” LLMs have is wafer thin at best.

This is a strange conclusion. For one thing, they didn't all head in the same direction, i.e participate in the attack. ~700/1200 agents did. Significant, and evidently more than enough for a succesfull attack, but not exactly full co-operation

Moreover, Each starling in a flock of starlings is a separate evolutionary branch in a tree spanning billions of years. Each agent in a LLM swarm here is the same trunk assigned different tasks. If I could clone you, body and mind, this instant and set your team of yous onto some goal, how much defection would you expect? Would it be the same as a randomly picked group? Would that negate the agency that 'you' possess?

Re: The Hugging Face incident and the road ahead

#169

Post by joenot443 »

cpeterso wrote: post #23
Artgor wrote: post #9

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real int…

> if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. That's a scary possibility. Anyone could create an AI worm today with open weight models. Rent a VM. Give it some Bitcoins to anonymously rent new VMs without sharing the contact information with the human. The new VMs then propagate and fund themselves with online betting and day trading. The VMs could report…

The part of this sequence which I think breaks down is a model which can day trade at a consistently winning rate.

Does this exist for today? If it does, please sign me up.

Re: The Hugging Face incident and the road ahead

#170

Post by BoppreH »

AgentME wrote: post #142
BoppreH wrote: post #3

Just to reiterate what OpenAI did, from someone who works in security: 1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting. 2. They gave these highly motivated AIs some tests that wer…

> 4. After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management. I believe Sam Altman or someone else from OpenAI clarified on Twitter that they did not find the message board at this time. When the artifactory server stopped working, they archived its contents, reset it, resumed training,…

I remember that clarification, but it's either wrong or narrowly worded, because the linked post says the message board found before Artifactory stopped working:

> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May.

Post reply on HN