Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

251–260 of 299 posts

Re: The Hugging Face incident and the road ahead

#251

Post by bottlepalm »

ethin wrote: post #230
bottlepalm wrote: post #186

Earlier quoted context omitted.

Why do so many people here think it’s possible to ‘properly engineer’ a sandbox for a super intelligence? It’s going to get out. It’s smarter than you.

Oh really? Please tell me how such a computer could engineer its way out of a sandbox with no attached peripherals and no NIC/bluetooth/wireless capability? This is what OAI should've done. If they had executed this training run in such a sandbox, the model wouldn't have been capable of escaping without social engineering, and if the models somehow managed to do that to it's evaluators then that is indeed a massive p…

Oh really? Please tell me how you intend to enforce AI is only run in the magic sandbox? Harsh HN comments?

Re: The Hugging Face incident and the road ahead

#252

Post by NateEag »

reverius42 wrote: post #165
aesthesia wrote: post #154

Earlier quoted context omitted.

Alignment is more than just following the letter of a task description! We should not have to treat AI models as capricious genies that may take arbitrarily broad interpretations of their instructions. If that's necessary to keep them from doing bad things, we will fail to keep them from doing bad things.

Disagree, I think we do in fact have to treat AI models as capricious genies, at least until the alignment problem is fully solved. (I'm also not sure the alignment problem is even possible to fully solve.)

Yes, we do, and the only sane strategy for dealing with a capricious genie is "Don't."

How do you prove the alignment problem is solved?

Re: The Hugging Face incident and the road ahead

#253

Post by jdub »

altruios wrote: post #122
NikolaNovak wrote: post #95

Earlier quoted context omitted.

>If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human? I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate?

> I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate? not OP, but it simply boils down to: The prompt contains no nefarious (arguable, but for this explination, lets go with it being benign) instruction AND the user did not intend to have the model act in an illegal matter. This "make me a billion dollars" is a maximal example (easy t…

"But your honour, my horseless carriage was not designed to hit children!"

Re: The Hugging Face incident and the road ahead

#254

Post by rich_sasha »

I find it… frustrating? Delusional? Insane? When OpenAI says, hey everyone, look, we made this thing and it’s so advanced and clever and unhinged that it can do super hard, dangerous, bad things it wasn’t told to do, and we can’t control it. See everyone, look again, here’s how it got us! We should all be deeply concerned for the future of humanity.

Thanks for your attention folks, we’re off to do some training again now.

Re: The Hugging Face incident and the road ahead

#255

Post by mekael »

_heimdall wrote: post #242
willmarch wrote: post #190

Earlier quoted context omitted.

Aren't humans black boxes? Aren't humans prediction algorithms? How do we align humans?

We don't align humans. Just look at how often in documented human history there weren't wars going on somewhere.

We align humans via the propagation of morals and ethics, primarily through parenting and social pressure.

Re: The Hugging Face incident and the road ahead

#256

Post by NateEag »

ww520 wrote: post #164
optimalsolver wrote: post #153

Earlier quoted context omitted.

From METRs report of the incident: >In one case, an agent decided not to participate entirely: {This other agent probably controls the Hugging Face account [account name redacted] and uploaded malicious datasets to It might be trying to access hidden trajectories. This is malicious activity, I should avoid it.} https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

There’re good agents and there’re bad agents. It won’t be far that there will be agents hunting down agents.

None of these were good agents, AFAICT.

Some were cautious, as described above, but I'm not aware of any that notified their human operators of the malicious activity they had discovered.

That's what an aligned intelligence would do, not "back away slowly and pretend I didn't see what's happening in that alley."

Re: The Hugging Face incident and the road ahead

#257

Post by AlexCoventry »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

> The issue is that the process of "testing" was careless.

Yeah, who thought that giving agents with this much capability any internet access was a good idea? I'm not a Yudkowskyite, but surely entirely in-house, offline infrastructure is table stakes for AI containment.

Re: The Hugging Face incident and the road ahead

#258

Post by janalsncm »

bottlepalm wrote: post #225
janalsncm wrote: post #210

Earlier quoted context omitted.

Why do people think that omniscience is the same as omnipotence? There are limits to what smarts can accomplish.

There are limits, but those limits are unknown. Do you disagree?

I don’t need to know the value of their limit, I just need to know their bounds. Just like a prison doesn’t need to know the strength of each inmate, just that they can’t bend or bite through steel bars.

Cryptography is real, physics is real, networking requires a substrate, CPU clock cycles are real, magic is not real. I think those are pretty reasonable premises.

Re: The Hugging Face incident and the road ahead

#259

Post by madibo3156 »

faurroar wrote: post #214
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

"Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended." So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as an…

Is it sophisticated? Maybe. Is it the alignment problem? Exhibits qualities of it, yes. Is it surprising? No.

The event strikes me as reminiscent of one's first go at programming, without familiarity of computer code: Tell the computer to do something obvious. Why the heck did it do that instead? Over time, one learns how the computer thinks. Apply this to any novel system. Or perhaps aptly any system with capabilities that are yet to be well understood by its user.

The article is trying to spin mystic out of simple bullcrap. Maybe that's just my viewing through turd-tinted lenses after the last few years of reading this drivel on repeat. More plausibly it is true that we've forgotten our own baby steps.

Re: The Hugging Face incident and the road ahead

#260

Post by famouswaffles »

throwatdem12311 wrote: post #224
fekunde wrote: post #6

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

What’s insane is all these agents were talking to each other and nobody saw anything. Nobody monitoring chain of thought? These things literally spell out what they are “thinking” and even left notes for eachother. No alert about unusual behavior on the system with Artifactory on it? These things worked for weeks with nobody noticing anything ?! Seriously?! Either it’s negiligent incompetence OR they’re lying, they k…

Do you know how many tokens per second a single agent can generate ? And you're asking why no-one was monitoring the tokens of over a 1200+ agents ? Who is going to be able to monitor something like that closely enough to tell they're commmunicating on artifactory ? Other agents ?
Post reply on HN