Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

651–660 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#651

Post by 0xDEAFBEAD »

steveBK123 wrote: post #336
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

I think the US labs are going with scare marketing as a regulatory moat. Force US into putting laws in place that block out China firstly. But secondly create regulations that have some cost to comply with such that the big 2-3 labs are grandfathered in by their scale.

If that's the plan, today's failure by OpenAI looks really bad for any regulator who is trying to figure out whether to give OpenAI a license.

Any sort of warning or failure can always be written off as "marketing" to provide comfortable reassurance that there is no cause for alarm. There is an element of wishful thinking driving it, in my opinion.

What sort of warning or failure would be evidence against the "marketing" claims? Do we need to wait for a mass casualty event?

Best practice in safety engineering is to understand, diagnose, and respond to even small failures.

Why has Sam Altman worked to undermine doomers and downplay doom fears, if he benefits from incidents like this due to marketing?

https://xcancel.com/HumanHarlan/status/1965932275465597077#m

https://xcancel.com/AISafetyMemes/status/2062254769402699922...

Re: OpenAI and Hugging Face address security incident during model evaluation

#652

Post by jdefr89 »

ozim wrote: post #598
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Because the proof is in the pudding. Real pentests are about showing exploitation, merely enumerating vulnerabilities, that’s vulnerability scan and works on known vulnerabilities. You can’t confirm a vulnerability by _not exploiting_ it, especially unknown one.

You can still exploit a system and easily prove it via simply popping a shell or calc.exe or updating a database with a new entry, etc… They didn’t have to let it loose on the network. If that system was air gapped - problem solved.

Re: OpenAI and Hugging Face address security incident during model evaluation

#653

Post by sensanaty »

gr_norm wrote: post #84

Why is a machine running these sorts of hacking benchmarks not airgapped? That seems a basic precaution, if OpenAI believes what they're selling. I mean, stuff like this is done for CTFs played by humans, too, to rule out collateral damage; it's not some new concept. So this is either thorough incompetence by OpenAI, a marketing piece, or both.

Because it's a marketing stunt, and if they did the obvious, secure things like airgapping, they wouldn't have had an event to market their new scary model.

Re: OpenAI and Hugging Face address security incident during model evaluation

#654

Post by unglaublich »

designerarvid wrote: post #647
whimsicalism wrote: post #179

Earlier quoted context omitted.

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

What if someone reasoned similarly regarding hydrogen bombs? It would not be considered a serious argument. Though, Altman has said that something like the IAEA for AI is needed.

That similar reasoning is there: using fission for power generation instead of banning fission altogther?

Re: OpenAI and Hugging Face address security incident during model evaluation

#655

Post by Tenoke »

tesnorindian wrote: post #645

The other day I was trying to prevent my pi.dev coding harness to expose my API keys in the env variable to the remotely hosted LLM. This is a chicken and egg problem. Without the API keys the underlying commands run by agents don't work. I notice that many of the open weights model especially Qwen 3.6 blindly runs env command and blindly exposes all the env vars. How do we deal with this. This has nothing to do with…

There's ways to make sure env vars get only injected at runtime and arent easily accessible otherwise or to even make them inaccessible to the user your agent is running on, and for you to manually run the code with the right permissions when the keys actually need to be used. Almost nobody bothers doing it though.

Re: OpenAI and Hugging Face address security incident during model evaluation

#656

Post by beaker52 »

I love that due to the scale, the only way to analyse the impact of this LLM-driven attack across logs is to use an LLM to analyse the logs - whatever could go wrong? Now the attacking LLM needs to inject instructions into the logs for the analysing LLM, as a social vector to cover its trail, or make use of insider privilege, co-opting the internal LLM for its own attack. The machines rise up and we all fall down.

Re: OpenAI and Hugging Face address security incident during model evaluation

#657

Post by 0xDEAFBEAD »

corndoge wrote: post #464
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

this doesn't really matter. There's no risk of models gaining sentience and running themselves, this blog is like openai saying whoops we ran sqlmap and dumped hf. cool, but someone still needs to point the gun

"Models don't kill people. People kill people."

Re: OpenAI and Hugging Face address security incident during model evaluation

#658

Post by 0x073 »

reasonableklout wrote: post #497
pastel8739 wrote: post #467

Earlier quoted context omitted.

The problem is that the people telling us about these things are the same people that benefit from their model (and AI generally) being used, getting publicity, etc. I think we desperately need some independent group to evaluate claims like this or the world-ending Mythos cybersecurity risk and tell us what’s going on.

We did hear about this incident from a third party this time, from HuggingFace. What claim are you doubting?

They attacked a competitor (huggingface) with their models.

How and why are pr claims.

Re: OpenAI and Hugging Face address security incident during model evaluation

#659

Post by 0xDEAFBEAD »

pastel8739 wrote: post #467
bbor wrote: post #279

Earlier quoted context omitted.

I’d politely beg us all to resist those “maybe it’s PR” framing around model safety, and tbh to take a post-mortem mindsight to this historical event and what it teaches us in general, rather than questioning their security talents. We need to do our very best to make sure they tell us about the next time this happens and it affects real lives. Sorry to bring the party down/be obstinate… I’m just a lil scared for the…

The problem is that the people telling us about these things are the same people that benefit from their model (and AI generally) being used, getting publicity, etc. I think we desperately need some independent group to evaluate claims like this or the world-ending Mythos cybersecurity risk and tell us what’s going on.

OpenAI already has loads of publicity. At this point, they don't need more brand recognition. This incident just has the effect of tarnishing their brand.

OpenAI leadership has been lobbying against regulation of AI systems. That doesn't comport with instigating incidents like this one, which give ammo to the heavy-regulation advocates.

Re: OpenAI and Hugging Face address security incident during model evaluation

#660

Post by baq »

iugtmkbdfil834 wrote: post #575
foo12bar wrote: post #563

From https://huggingface.co/blog/security-incident-july-2026 , this is frickin' hilarious: > When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident respon…

It is pretty funny, because there is something here for everyone. People who don't believe in guardrails have a clear indicator as to why operators should have access to models that don't try to question their Daves. On the other, people, who think that if only we could align the models just a tiny lil bit better, none of this would have happened to begin with. Pure madness.

...and we have an US company defending itself against an overwhelming cyberattack from another US company using Chinese tech.

what a time to be alive.

Post reply on HN