Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

371–380 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#371

Post by conradkay »

0x5FC3 wrote: post #177

0days ending in RCE (multiple!) for presumably closed source software are for the lack of a better phrase, labour of love. You run the exact same versions running on the target, blackbox test, fuzz it, craft an exploit, test, perfect it. For exploits which are of the memory kind, hook it to a debugger, decompile and what not. The exploits mentioned here seem to be code execution directly while processing input. Huggi…

I find it trustworthy since we had Hugging Face's account first: https://huggingface.co/blog/security-incident-july-2026

I don't think they have any real motive to shill OpenAI, probably closer to the opposite since they're so involved in open weights

Re: OpenAI and Hugging Face address security incident during model evaluation

#372

Post by janalsncm »

NyxWulf wrote: post #210
janalsncm wrote: post #186

Absolutely bewildering. If I am building a giant cannon and blow a hole straight through my neighbor’s house, I’m not going to say “we are working with our neighbors to improve their giant cannon defenses”. OpenAI brought this weapon and as far as I’m concerned they used it on another party. Morally it probably matters that this happens because they don’t know how their weapon works. Legally I always thought it was i…

It's an interesting point, but this is more like we are building a giant autonomous canon, that escaped the lab, the testing range, defeated state of the art and serious security protocols, and then blew a hole in the neighbors house. Our legal and philosophical perspectives are deeply rooted in humans being the actors. Doing that in a residential home is unforgiveable. Doing it responsibly on a military range is exp…

Yes, the human actors in your scenario were the ones who built the autonomous cannon and turned it on while knowing that 1) a good neighbor does not destroy their neighbor’s property 2) cannons can destroy property.

Also OpenAI specifically turned off their own cybersecurity guardrails to run this experiment. In other words it was able to escape the lab specifically because they turned them off. A human made the choice to turn off the guardrails.

Re: OpenAI and Hugging Face address security incident during model evaluation

#373

Post by catigula »

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

The problem is that it’s impossible to out think a robot you designed to be an expert at cybersecurity on the topic of cybersecurity. The alternative is not developing this and that’s not going to happen.

Re: OpenAI and Hugging Face address security incident during model evaluation

#374

Post by pixl97 »

jrflo wrote: post #305
zmj wrote: post #289

Earlier quoted context omitted.

It wasn't. The model discovered and exploited a vulnerability in their package manager proxy to (inferred) move laterally through their internal systems to one with open internet access.

That's not what airgapped means. Airgapping means the model exists on a system where there is no ethernet cable plugged in to a router or wifi card installed, it is physically impossible for it to access the internet because the hardware connection does not exist. If it was able to get on the internet, it was not airgapped.

And when it tricks on of the researchers to move data across the gap for them?

Long before LLMs existed we already knew that a sufficiently intelligent agent, human or otherwise, is not stopped by air gaps. The relatively weak models we have now can already figure out when their tested and cut off from the internet and change their behavior.

Re: OpenAI and Hugging Face address security incident during model evaluation

#375

Post by famouswaffles »

jscd wrote: post #196
blovescoffee wrote: post #75

Earlier quoted context omitted.

Huggingface literally reported the outage separately and did not know who caused it at first.

Does that change anything? We're still relying on OpenAI's account of where the LLM was running, what sandboxing restrictions were in place, the task it was given, etc. Even assuming they're telling the truth about what this LLM's goal was, they still have motivation to be less than honest about the state of their "highly isolated environment." Either this model was really operating in a truly locked down intranet an…

It doesn't matter much whether they are lying about how secure the environment is when they train and ship these models for other actors. Believe it or not, they're not spending hundreds of millions training these models for only themselves. Hacking Hugging face is an achievement on its own and the important relevation here.

Re: OpenAI and Hugging Face address security incident during model evaluation

#376

Post by elictronic »

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

A few hundred billion to pretend you have AGI. I'm going with fraud personally but at the end of the day the current admin is incentivized to do nothing.

Re: OpenAI and Hugging Face address security incident during model evaluation

#377

Post by JaRail »

nrmitchi wrote: post #362

If you are attempting to run exercises like this, it is wildly negligent to not be running it in a physically-airgapped environment (potentially with a physical power shutdown). You can not tell me that OpenAI doesn’t have the resources or ability to run tests like this in a physically-non-networked environment w/ sufficient compute for its needs.

If it was just one test, sure. But if they're spinning these up continuously with new models on tens of thousands of GPUs, air gapping becomes impractical. I would mostly fault them on having no guardrails at all. They should have a monitor/external harness that looks for successful access to external networks then stop it there. They may as well let the models test their own networks for vulnerabilities. That's going to be really important to have going forward.

Re: OpenAI and Hugging Face address security incident during model evaluation

#378

Post by neitherboosh »

JumpCrisscross wrote: post #191
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

> Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? Because we continue to have zero evidence that aligment is an actual risk.

What would compelling evidence look like to you?

Re: OpenAI and Hugging Face address security incident during model evaluation

#379

Post by stevenhuang »

georgespencer wrote: post #300

> We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly. All the AI in the world and they still can't write.

Reads fine to me.

Re: OpenAI and Hugging Face address security incident during model evaluation

#380

Post by pixl97 »

karmasimida wrote: post #174
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Because the model capability is beyond their expectation. This is brilliant marketing but I think it is real.

I mean we already see models exploit people's misunderstanding of how Docker works to get root without using su. And if you are one of the lucky people in cyber security that has been given a fat stack of tokens by the model providers you get to see some pretty wild exploit chains get put together by the models. Models are much better at detecting insecure code than writing actual secure code at this point.
Post reply on HN