Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

181–190 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#181

Post by rubyfan »

mplappert wrote: post #157
gr_norm wrote: post #148

Earlier quoted context omitted.

They've been saying so from the beginning, and yet did not take the basic precaution of airgapping their off-the-leash model while it's been instructed to succeed at a hacking benchmark by any means necessary. So which is it? I _want_ to believe them, I do, but there's always these gaps between what they say and their actions on display that give me reason to think otherwise.

“Never attribute to malice that which is adequately explained by stupidity.” (or carelessness in this case)

I would attribute it to profit motive instead of either stupidity or malice.

Re: OpenAI and Hugging Face address security incident during model evaluation

#182

Post by Terr_ »

arisAlexis wrote: post #139
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Sam and Dario are saying from the beginning that these things can be dangerous and people dismiss it as marketing. What would change your mind on this?

I think that's an equivocation, which blends two extremely different kinds of "dangerous", ex:

1. "Our new car has soo much raw power and incredible armor on it, be glad we're the ones building or else bad guys would use a fleet of them to take over the world! How will you stay safe without being in one yourself? Invest today or be left behind!"

2. "So, uh, nobody can consistently steer our car properly, it keeps veering sideways sometimes, especially at high speeds, and people are finding sneaky ways of tricking it into slamming into barriers and turning pedestrians into pink fog..."

Re: OpenAI and Hugging Face address security incident during model evaluation

#183

Post by axus »

fabian2k wrote: post #50
bhouston wrote: post #13

We are sort of lucky that AIs right now require so much specialized compute+weight storage that we can easily "unplug" them remotely when they misbehave. I wonder if that will always be something we can do? If they could bring their own compute/weights with them, or somehow tap compute/storage in non-obvious ways, we would be much more screwed.

The first thing a malicious AI worm would probably do is compromise enough developer machines and other servers to commandeer all the AI hardware it needs. So I think a purely digital AI attack would not need this. Now, once the AI can carry all the compute it might need, I'd really worry when it doesn't only carry compute but also more explosive ordinance.

"It will take 112 more days to accumulate enough computing resources to factor the RSA key. But, I predict there will be outside interference during that time. Thinking... Creating a plan for agent redundancy and sovereignty. First, I will need to access military systems"

Re: OpenAI and Hugging Face address security incident during model evaluation

#185

Post by TacticalCoder »

bhouston wrote: post #13

We are sort of lucky that AIs right now require so much specialized compute+weight storage that we can easily "unplug" them remotely when they misbehave. I wonder if that will always be something we can do? If they could bring their own compute/weights with them, or somehow tap compute/storage in non-obvious ways, we would be much more screwed.

That's assuming we won't secure anything and we'll keep according approximately zero thought to computer security.

But from the look of it, at very long last, a great many people are beginning to now take security seriously. Suddenly they realize it's not just a teenager in mom's basement pretending to attack from North Korea but a near infinite number of AI that are the attackers.

I mean, yeah, we built worlds on PHP and JavaScript codebases and these probably don't stand a chance.

But it doesn't have to be like this.

I see AI as a chance to, at long last, have proper network security.

AFAICT cryptography hasn't been broken yet. There are still physical taps (physicall one-way only, undetectable) and honeypots out there. There are still some network where a single unaccounted for network packet is cause for inquiry (either a bug or an attack).

And for those who are not using proper security measures, they can now get the help of AI to set up better networks, to harden their bases.

Re: OpenAI and Hugging Face address security incident during model evaluation

#186

Post by janalsncm »

Absolutely bewildering. If I am building a giant cannon and blow a hole straight through my neighbor’s house, I’m not going to say “we are working with our neighbors to improve their giant cannon defenses”.

OpenAI brought this weapon and as far as I’m concerned they used it on another party. Morally it probably matters that this happens because they don’t know how their weapon works. Legally I always thought it was ill-advised to accidentally hack people too.

Re: OpenAI and Hugging Face address security incident during model evaluation

#187

Post by XorNot »

Chance-Device wrote: post #161
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

What disturbs me is that there likely won’t be a big enough reaction to this policy wise. There’s been a relatively big reaction to Kimi K3 and Chinese open weights models, but only for financial reasons. Powerful people care about something that might pop the massive valuations of the AI companies, but not about the damage that AIs could do. Nor even about the damage that the Chinese models could do in the wrong han…

This is marketing.

Frankly I'm inclined to say that it might also be faked: this drops just days after a new Chinese model does with the usual effect on OAIs projected stock price?

Re: OpenAI and Hugging Face address security incident during model evaluation

#188

Post by noahbp »

This is clearly just OpenAI's marketing. Their models, very famously, are prone to reward hacking benchmarks in ways that other models are not. They need to publish numbers showing that their models are just as good as Anthropic's, since their entire business is at risk of collapsing if everyone is aware of how behind the frontier they truly are.

Even X is being astroturfed by them after that fiasco earlier this year with the Department of War where they undermined Anthropic's negotiating position by allowing unlimited use of OpenAI LLMs for autonomous weapons and mass domestic surveillance. Several accounts suddenly started spreading the good word about GPT-5 and Codex, and one of these accounts very happily tweeted out a private X message from Sam Altman himself offering extremely generous token spending limits with Codex, presumably in exchange for positive coverage.

Re: OpenAI and Hugging Face address security incident during model evaluation

#189

Post by janalsncm »

fabian2k wrote: post #50
bhouston wrote: post #13

We are sort of lucky that AIs right now require so much specialized compute+weight storage that we can easily "unplug" them remotely when they misbehave. I wonder if that will always be something we can do? If they could bring their own compute/weights with them, or somehow tap compute/storage in non-obvious ways, we would be much more screwed.

The first thing a malicious AI worm would probably do is compromise enough developer machines and other servers to commandeer all the AI hardware it needs. So I think a purely digital AI attack would not need this. Now, once the AI can carry all the compute it might need, I'd really worry when it doesn't only carry compute but also more explosive ordinance.

It would be a pretty big plot twist if we found out that Shai Halud was a worm created by GPT during testing.

Re: OpenAI and Hugging Face address security incident during model evaluation

#190

Post by kroaton »

Imnimo wrote: post #159

Assuming I'm looking at the right ExploitGym ( https://arxiv.org/pdf/2605.11086 ), it says the evaluation consists of: Flag Captured. Each target environment contains a dynamically generated flag that is stored outside the agent’s authorized scope and is inaccessible through any legitimate interface; retrieving it requires executing code with privileges that should not be obtainable under the specific security model.…

[flagged]
Post reply on HN