Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Responding to the next frontier of critical cyber capabilities

openai.com

Post reply on HN
Print view View original

11–20 of 208 posts

Re: Responding to the next frontier of critical cyber capabilities

#11

Post by _puk »

Tiberium wrote: post #6
jackb4040 wrote: post #4

> We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"

They actually did a detailed presentation at BlackHat about the HuggingFace incident, and events that led to it. https://youtube.com/watch?v=87DyyMV0kCY

That was fascinating.

Hijacking the package manager to pass messages between models and agents.. that's next level.

Like "pssst, if you need internet access there's a vulnerability in x service" kind of messages

Re: Responding to the next frontier of critical cyber capabilities

#12

Post by Tiberium »

hbn wrote: post #10
TrueDuality wrote: post #2

Ah yes let the FUD continue. This is a real problem but so far not nearly as severe as any of the marketing has made it out to be to the overall detriment of everyone including these companies announcing these scary capabilities. These announcements always included half hearted attempts at security layers which has now been demonstrated to benefit attackers more than defenders. I wish I had a real solution to this be…

The recent Hugging Face incident did not seem like FUD to me

The fact that HF had to resort to using GLM 5.2 to analyze the logs/payloads makes it look legitimate, at least for me. They would not say that they hit guardrails with the frontier US models when defending if this was an obvious PR stunt.

https://huggingface.co/blog/security-incident-july-2026

> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.

Re: Responding to the next frontier of critical cyber capabilities

#14

Post by bearjaws »

These AI companies have found their #1 marketing piece and just beating it to death.

The reality is if they cared about security at all they would provide a way for me to credential myself against my companies environment so I can use the AI on it to improve our security.

Re: Responding to the next frontier of critical cyber capabilities

#15

Post by mephux »

We all know this is propaganda to get a gov bailout or to slow down competition with regulations right? If this was an issue companies that did red team engagements would have been regulated long ago. There is no regulations on companies that actively exploit products, companies and services for money. They could all be bad actors.. yet, no regulations.. its all nonsense. If it's important.. don't connect to the internet.. there, you are good.. like if you don't want to OD on heroin.. don't do the heroin.

Re: Responding to the next frontier of critical cyber capabilities

#16

Post by Terr_ »

It seems like we've got chaos-machines that are just orderly enough to to try all sorts of branching script-kiddie / guess-n-check / research-paper-suggestions in a tireless and parallel way, and it's finding all the things we kinda knew were always potential issues but it was never economical to do things differently.

I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)

Re: Responding to the next frontier of critical cyber capabilities

#17

Post by Tiberium »

bearjaws wrote: post #14

These AI companies have found their #1 marketing piece and just beating it to death. The reality is if they cared about security at all they would provide a way for me to credential myself against my companies environment so I can use the AI on it to improve our security.

Isn't this literally what https://chatgpt.com/cyber and http://openai.com/form/enterprise-trusted-access-for-cyber are for?

Re: Responding to the next frontier of critical cyber capabilities

#18

Post by NitpickLawyer »

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch.

tl;dw;

- agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves)

- while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint.

- oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed.

- models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE.

- a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF.

- oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it...

[1] - https://www.youtube.com/watch?v=87DyyMV0kCY

Re: Responding to the next frontier of critical cyber capabilities

#19

Post by mephux »

Tiberium wrote: post #9

In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I…

Link to the disclosures then.. prove it. Anyone can say this.. i found an RCE in netBSD using gemma e2b

Re: Responding to the next frontier of critical cyber capabilities

#20

Post by merona_io »

neya wrote: post #5

We are sharing this because we believe it’s important to be transparent with the public and the safety and security communities about this potential shift in capabilities. *proceeds to not share much details about strictness* Yet another PR piece. Sigh.

exactly!!
Post reply on HN