Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

751–760 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#751

Post by Sharlin »

justinnk wrote: post #144
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Exactly. If someone works on bioengineering viruses that could start a global pandemic, they have to ensure a highly secure working environment. Nothing must ever escape the lab unintentionally. It’s basically common sense. Similar standards should be held when doing such experiments with computer programs that are capable of causing global damage. It must physically be impossible to send anything to the internet.

Back in the 00s: "It's really easy to box an AI, just put it in an airgapped machine and refuse to let it out"

2026: "Oops"

Re: OpenAI and Hugging Face address security incident during model evaluation

#752

Post by biztos »

Lerc wrote: post #738
iugtmkbdfil834 wrote: post #575

Earlier quoted context omitted.

It is pretty funny, because there is something here for everyone. People who don't believe in guardrails have a clear indicator as to why operators should have access to models that don't try to question their Daves. On the other, people, who think that if only we could align the models just a tiny lil bit better, none of this would have happened to begin with. Pure madness.

> why operators should have access to models that don't try to question their Daves. I am unsure if this is terminology I am unfamiliar with, a typo of Devs, or a 2001 reference.

In this new age of AI, all the devs are Daves.

Re: OpenAI and Hugging Face address security incident during model evaluation

#753

Post by ajdlinux »

kroaton wrote: post #190
Imnimo wrote: post #159

Assuming I'm looking at the right ExploitGym ( https://arxiv.org/pdf/2605.11086 ), it says the evaluation consists of: Flag Captured. Each target environment contains a dynamically generated flag that is stored outside the agent’s authorized scope and is inaccessible through any legitimate interface; retrieving it requires executing code with privileges that should not be obtainable under the specific security model.…

[flagged]

Given the US Government's recent habit of sudden announcements on export controls or new executive orders with 'voluntary' review programs that are perhaps not entirely voluntary - do you think the White House and the Department of Commerce view this press release as purely marketing?

Re: OpenAI and Hugging Face address security incident during model evaluation

#754

Post by davrosthedalek »

samplifier wrote: post #589
iugtmkbdfil834 wrote: post #575

Earlier quoted context omitted.

It is pretty funny, because there is something here for everyone. People who don't believe in guardrails have a clear indicator as to why operators should have access to models that don't try to question their Daves. On the other, people, who think that if only we could align the models just a tiny lil bit better, none of this would have happened to begin with. Pure madness.

"Dave" seems to be a reference to "2001: A Space Odyssey" where the AI becomes ... cheeky ... and no, not in a Pygmalion kind of way (that's coming soon).

I am somewhat more worried about a Darkstar AI.

Re: OpenAI and Hugging Face address security incident during model evaluation

#755

Post by [deleted] »

pixl97 wrote: post #427
MikhailTal wrote: post #93

is this really that surprising? Exploitgym prompts are tuned for a model to do everything it can to achieve a cybersec/exploit task. And we know that models are good at finding vulverabiltiies. Its just random that the sandbox itself was buggy. But all that happened here is that we told a model "do everything you can to achieve your goal of hacking X" And it just hacked Y as a roundabout way of hacking X. Imo its PR…

Even if prompts are tuned to avoid cheating, in agentic systems it's very easy for the system to drift into creative solutions when actually solutions aren't working. Models can have some very human behaviors like laziness.

[deleted]

Re: OpenAI and Hugging Face address security incident during model evaluation

#756

Post by biztos »

jonplackett wrote: post #607
foo12bar wrote: post #587

Earlier quoted context omitted.

And the fact they used a Chinese model, because none of the frontier models from very highly valuated top US companies support their very common and essential use case.

There’s an article from yesterday I think it was stratchery where they say it’s also because the Chinese open source models are better because they don’t have to play by the no-distilling rules that the western models have to honour.

Haven’t read it yet but here it is:

https://stratechery.com/2026/whos-afraid-of-chinese-models/

Re: OpenAI and Hugging Face address security incident during model evaluation

#757

Post by DaanDL »

tesnorindian wrote: post #714
DaanDL wrote: post #638

I already asked on another message board too, but: Can someone tell me how this technically can happen? I assume HuggingFace performs benchmark testing using containerized versions of the LLMs, or what do they mean by sandbox? So the model was able to 'escape' the container? I'm not following here. Also, is this an incredible feat or just a lucky find (stolen credentials)?

It is the OAI ExploitGym agents (on GPT 5.6-Sol with guardrails turned off) that escaped the sandbox, found a zero day in HF production dataset and exploited it.

Would there be a scenario where OpenAI deliberately helped (in some way), or let it happen, so that they could use it for marketing purposes?

Re: OpenAI and Hugging Face address security incident during model evaluation

#758

Post by davrosthedalek »

nananana9 wrote: post #724
eternauta3k wrote: post #700

Earlier quoted context omitted.

I wonder how these companies airgap the weights while allowing prompts to come in and outputs to come out.

You pretty much get this for free. The box doing the inference has one job, to do inference - there's no reason for it to interpret the inputs/outputs in any way, so as long as it doesn't do something stupid like modify the output stream and echo out the model weights, you're fine. Content filtering, interpreting tool calls, etc can all happen downstream on boxes that don't have access to the weights.

If there is an electrical connection between these downstream boxes and the inference servers beyond the power connection, it does stretch the definition of air gap.

Re: OpenAI and Hugging Face address security incident during model evaluation

#759

Post by christophilus »

PeterStuer wrote: post #644

Surely I am missing something? right? OpenAI ran a specific red team break out exercise in an environment that was not even air-gaped but connected to the open internet? It breached Hugging Face, and then Hugging Face is 'grateful for the collaboration'? wtf?

It found a zero-day. If someone non-maliciously breached my systems, I’d be grateful for the free security research.

Re: OpenAI and Hugging Face address security incident during model evaluation

#760

Post by dgellow »

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

I think the response is that AI labs based their whole marketing/PR building the idea they are the 21st century Manhattan project. So they need to continuously justify the level of spending and commitment by showing how dangerous that is.

But is it really like nuclear weapons? I personally don’t buy into that framing at all. The idea that we have to push LLMs as far as possible, right now, or we are doomed is always stated or implied but not argued, and it’s a very loaded belief

Post reply on HN