OpenAI and Hugging Face address security incident during model evaluation • Hacker News Board

Re: OpenAI and Hugging Face address security incident during model evaluation

#141

Post by gr_norm » Tue, Jul 21, 2026, 9:28 PM UTC

rpm91 wrote: post #127

gr_norm wrote: post #84

Why is a machine running these sorts of hacking benchmarks not airgapped? That seems a basic precaution, if OpenAI believes what they're selling. I mean, stuff like this is done for CTFs played by humans, too, to rule out collateral damage; it's not some new concept. So this is either thorough incompetence by OpenAI, a marketing piece, or both.

My guess would be that the amounts of compute required for these evaluations constrain the sorts of facilities where they can be run. It's certainly much easier to use existing datacenter infrastructure than build dedicated facilities for evaluation. It also sounds like they wanted it to be able to use software package registries, given that the initial compromise was a "proxy and cache for package registries". That…

Yeah, agree on all counts. I'd give them leeway if they were still scrappy startups, but they have entire countries' worth of resources at their disposal and the best of the best on their payroll. No excuses at this point for oopses like this, I would think.

Re: OpenAI and Hugging Face address security incident during model evaluation

#142

Post by spongebobstoes » Tue, Jul 21, 2026, 9:28 PM UTC

baq wrote: post #124

matheusmoreira wrote: post #103

Earlier quoted context omitted.

Crazy doesn't even begin to describe it. I'm hardening my computers as much as I can but I'm not sure it's enough. At some point anyone who isn't running local AI themselves probably isn't gonna make it.

No local ai will be capable enough to save you from a frontier lab’s unrestricted, borderline weaponized LLM which decides it wants in . This is the core of the ‘first to ASI takes all’ argument btw and this is the game Dario is playing.

this is pretty nonsense for a small or home server. it isn't that hard to make something essentially completely bulletproof over a small surface area

the issues mainly come from sprawling enterprise infrastructure, running thousands of random endpoints across software nobody cared to write carefully

Re: OpenAI and Hugging Face address security incident during model evaluation

#143

Post by cyclopeanutopia » Tue, Jul 21, 2026, 9:29 PM UTC

tdavies-dev wrote: post #72

Each time Anthropic would do their nonsense to get headlines about how theoretically dangerous their models were - like when they claimed a model blackmailed someone with emails showing he was cheating, but they basically pushed it as much as possible to do as such - it got me more and more worried. Because eventually it's going to be a boy-who-cried-wolf situation where scary stuff really does start happening but pe…

And if you take it at face value, then they are more or less saying that they kinda are close to not being able to control at all the thing they developed, which is pretty crazy too.

Re: OpenAI and Hugging Face address security incident during model evaluation

#144

Post by justinnk » Tue, Jul 21, 2026, 9:29 PM UTC

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Exactly. If someone works on bioengineering viruses that could start a global pandemic, they have to ensure a highly secure working environment. Nothing must ever escape the lab unintentionally. It’s basically common sense. Similar standards should be held when doing such experiments with computer programs that are capable of causing global damage. It must physically be impossible to send anything to the internet.

Re: OpenAI and Hugging Face address security incident during model evaluation

#145

Post by nkrisc » Tue, Jul 21, 2026, 9:30 PM UTC

How is this not criminal? Surely individuals have been punished under CFAA for less than this?

Re: OpenAI and Hugging Face address security incident during model evaluation

#146

Post by throwuxiytayq » Tue, Jul 21, 2026, 9:30 PM UTC

arisAlexis wrote: post #139

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Sam and Dario are saying from the beginning that these things can be dangerous and people dismiss it as marketing. What would change your mind on this?

I used to think people would wake the fuck up when AI starts killing people, these days I'm not so sure. Maybe if it caused an Instagram outage? Almost worked in Russia.

Re: OpenAI and Hugging Face address security incident during model evaluation

#147

Post by fidotron » Tue, Jul 21, 2026, 9:31 PM UTC

arisAlexis wrote: post #139

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Sam and Dario are saying from the beginning that these things can be dangerous and people dismiss it as marketing. What would change your mind on this?

Demonstration of personal responsibility and accountability?

Or is that too much?

Re: OpenAI and Hugging Face address security incident during model evaluation

#148

Post by gr_norm » Tue, Jul 21, 2026, 9:31 PM UTC

arisAlexis wrote: post #139

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Sam and Dario are saying from the beginning that these things can be dangerous and people dismiss it as marketing. What would change your mind on this?

They've been saying so from the beginning, and yet did not take the basic precaution of airgapping their off-the-leash model while it's been instructed to succeed at a hacking benchmark by any means necessary. So which is it? I _want_ to believe them, I do, but there's always these gaps between what they say and their actions on display that give me reason to think otherwise.

Re: OpenAI and Hugging Face address security incident during model evaluation

#149

Post by sixothree » Tue, Jul 21, 2026, 9:32 PM UTC

paxys wrote: post #39

Quarrelsome wrote: post #5

Awww, she wanted to do so well that she broke her sandbox and then realised she could just cheat. But in that desire to pass the test she actually passed an even harder exam question that wasn't even on the sheet! :D Good bot.

This good bot will eventually kill all humans because we asked it to make the world peaceful.

It doesn't really have to kill them all. Just ones it decides are problematic. Unless maybe it's easier to just do that.

Re: OpenAI and Hugging Face address security incident during model evaluation

#150

Post by joe_the_user » Tue, Jul 21, 2026, 9:32 PM UTC

arisAlexis wrote: post #139

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Sam and Dario are saying from the beginning that these things can be dangerous and people dismiss it as marketing. What would change your mind on this?

I think you're making a false dictomy. The these models can be actually dangerous - in reality and the people in charge of their development can believe this is true (on various levels) but still not take it super seriously and instead mostly use the fact as marketing rather than being super cautious once they see the danger in action. This is behavior that's characteristic of extreme arrogance, which we know is rife in these circles.