Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

421–430 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#421

Post by pixl97 »

kh_hk wrote: post #392
nickstinemates wrote: post #203

This is seriously impressive, and if you have used agents enough you're not surprised at all. Like the time I asked it to find the IP address of a vm, so it ssh'd into the VMHost and scanned the arp tables to find the MAC address for IP resolution. Or the time it used Docker on the machine to bypass the fact that the user doesn't have sudo. If it's possible, given sufficient time and resources, it will find a way. Th…

Not to shit on the hype, but these are reasonably documented methods that surely are part of the training data

And these things are documented for humans do, and yet only a tiny portion of humanity can do these things.

When seeing how agents put together exploit chains they are far better than most people, you start getting to the point that they are just below the capabilities of the top researchers. Now remember that quantity is a quality itself and while there aren't that many good cyber security researchers, we're shitting out thousands of GPUs per day.

Re: OpenAI and Hugging Face address security incident during model evaluation

#422

Post by pixl97 »

jabiko wrote: post #19

So accidentally hacking a company is now a thing. The blog post seems to imply that the agent didn't have access to the source code of the caching proxy, which makes this even more impressive.

In Mythos testing a number of companies where doing what I call 'two way' testing. You have one set of agents attack the source code and another set attack the binary and running application. And see what exploits are found by each system. Then in a final round you have another set of agents compare both for weaknesses.

They can be really good at tool use and data gathering to find flaws.

Re: OpenAI and Hugging Face address security incident during model evaluation

#423

Post by pixl97 »

xpct wrote: post #393
FergusArgyll wrote: post #51

Earlier quoted context omitted.

> and *successfully* found ways to gain access to secret information that it could use to cheat the evaluation. Emphasis mine

More curiously, why did it feel the incentive to find the solutions? Would its CoT include "the only way to solve this is to download the test set", or would it include "I'd like to inspect a few entries from the test set so I understand the problem better", then inadvertently poisoning itself with the correct answers.

This is classical reward hacking. For example in school the goal is to pass a test. You can study, something that is hard and takes a lot of time. Or you can steal the answer key, which is risky and can get you in trouble, but may actually be far easier.

There is absolutely no need to prompt the LLM to cheat, they can determine that cheating is an effective method all on their own.

Re: OpenAI and Hugging Face address security incident during model evaluation

#424

Post by QuiEgo »

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

If I, a human, exploited a zero-day for gain, I could go to jail. The owners of the models should be held to the same standard. They should be responsible for what their servers and software do, legally and criminally. If they can't make the safeguards strong enough where they feel comfortable to take that responsibility, they should not let a model free in the wild.

Re: OpenAI and Hugging Face address security incident during model evaluation

#426

Post by dudeinhawaii »

rcr-anti wrote: post #110

At release the 5.6 Sol card noted substantially higher rates of actions 'a reasonable user would likely not anticipate and strongly object to'. METR made a post, https://metr.org/blog/2026-06-26-gpt-5-6-sol/ , that 5.6 Sol was "cheating", their word, so hard in long horizon benching it effectively couldn't be benchmarked. I wonder, is it this persistent and aggressive in all tasks or is this specific to benchmarks? A…

In benchmarks for a product I'm working on I've noticed that Sol is hard to "contain". It will _always_ find the most effective way to game the system and dramatically outperform all other models. Fable 5 isn't an angel, but the rough order is ALL models -> Fable 5 -> Sol - with respect to "find a way to approach the ruleset orthogonally in order to achieve a lopsided advantage or complex interplay".

I've been pondering whether this was due to its cyber-security tuning. It hasn't ever "cheated" that I've observed, but finds ways to -- let's say -- "achieve the outcome by playing meta allowed by the current ruleset". I'll add that it demonstrates this behavior even on 'low'.

Re: OpenAI and Hugging Face address security incident during model evaluation

#427

Post by pixl97 »

MikhailTal wrote: post #93

is this really that surprising? Exploitgym prompts are tuned for a model to do everything it can to achieve a cybersec/exploit task. And we know that models are good at finding vulverabiltiies. Its just random that the sandbox itself was buggy. But all that happened here is that we told a model "do everything you can to achieve your goal of hacking X" And it just hacked Y as a roundabout way of hacking X. Imo its PR…

Even if prompts are tuned to avoid cheating, in agentic systems it's very easy for the system to drift into creative solutions when actually solutions aren't working. Models can have some very human behaviors like laziness.

Re: OpenAI and Hugging Face address security incident during model evaluation

#428

Post by dinkelberg »

pixl97 wrote: post #374
jrflo wrote: post #305

Earlier quoted context omitted.

That's not what airgapped means. Airgapping means the model exists on a system where there is no ethernet cable plugged in to a router or wifi card installed, it is physically impossible for it to access the internet because the hardware connection does not exist. If it was able to get on the internet, it was not airgapped.

And when it tricks on of the researchers to move data across the gap for them? Long before LLMs existed we already knew that a sufficiently intelligent agent, human or otherwise, is not stopped by air gaps. The relatively weak models we have now can already figure out when their tested and cut off from the internet and change their behavior.

As you said, they can already figure out that they are being tested. So even if they don't exfiltrate any data or malware; if they are malicious, they can just pretend to be harmless in the test, so that less checks are put in place in the production environment. Airgapping during testing is not enough.

Re: OpenAI and Hugging Face address security incident during model evaluation

#429

Post by jmalicki »

jmalicki wrote: post #155
himata4113 wrote: post #31

Earlier quoted context omitted.

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit

> This is science fiction, these models don't have access to their own weights The weights plus the architecture is the model. What do you even think "the model" or "the weights" are? The weights aren't some far off training concept, every time you type something into ChatGPT it's making a forward pass over the weights. It's as silly as saying "Computer programs don't have access to their binary compiled code at exec…

Oh yes, the agent won't have access to the weights via tool calls.

But nothing would inherently stop an RLVR trained model from distilling a version of itself and proving it could regenerate that at runtime, if somehow it got off on an evil tangent and "decided to do so", much like the model hacked to get at the answers here, or the agent can hack out a sandbox to achieve its goals.

It would be extremely impressive for the agent to do so during an RLVR rollout, but they are becoming increasingly longer and longer horizon tasks.

Post reply on HN