Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Be skeptical of OpenAI's rogue hacker agent story

theguardian.com

Post reply on HN
Print view View original

191–200 of 321 posts

Re: Be skeptical of OpenAI's rogue hacker agent story

#191

Post by kalkin »

why_at wrote: post #184
ACCount37 wrote: post #94

By now, I'm pretty confident that some people would keep screeching "it's just a marketing stunt, AI capabilities and AI risks aren't real, they're just doing this to prop up their stocks" even if they find a Cyberdyne Systems T-800 armed with a shotgun breaking down their front door. "It's a marketing stunt" is just denial trying to look like it's being clever.

I think you are conflating skepticism about AI companies' motives with skepticism about their capabilities. Even if OpenAI is being 100% honest in their reporting on this it's still good marketing for them. The fact that this outcome is good for their business and stock price makes me suspicious about how much this was a complete accident vs an "accident" that they allowed to happen by setting up the right environmen…

> this outcome is good for their business and stock price

You state this as fact--how do you know it? OpenAI isn't publicly traded. On Hiive OpenAI is marginally down in July: https://www.hiive.com/securities/openai-stock

Re: Be skeptical of OpenAI's rogue hacker agent story

#192

Post by yallpendantools »

jackb4040 wrote: post #90
dumberquestions wrote: post #24

There are some reasons the story could be inaccurate in some ways: OAI stands to benefit if people think their models are strong, and they have a history of doing things with dubious ethics (e.g. using data for training against the terms of its creators, abandoning the non profit mission, stealing or attempting to steal Apple IP). But there are also reasons why the story could be true: OAI are admitting that they app…

Investors have rewarded every story of "our models are too powerful to be controlled" since before ChatGPT. Let's stop pretending there is any real financial risk to OpenAI from events of this type. "Alignment research" is a sub-percentage-point fig leaf for them like the solar division at an oil company.

> Investors have rewarded every story of "our models are too powerful to be controlled" since before ChatGPT.

Can you elaborate/refresh my memory? IIRC before ChatGPT 3 there wasn't really an investor market for AI models, rather crypto. I remember playing with the likes of Stable Diffusion pre-ChatGPT 3 but only the likes of Altman and Musk were talking about AI too powerful to be controlled (which is notably the rationale of OpenAI's original charter); still investors in general were hardly invested at the time.

Re: Be skeptical of OpenAI's rogue hacker agent story

#193

Post by modeless »

trouve_search wrote: post #144
nikcub wrote: post #126

Earlier quoted context omitted.

> AI managed to escape using standard and well documented script kiddie methods > AI broke in using standard script kiddie methods. I've spent time gathering the detail of what happen here and while there are some solid theories and indicators, absolutely nothing so far has suggested a sandbox escape using "well documented script kiddie methods" or that the method used to break into the HF network was similar. Where…

From my reading, the sandbox escape came from the JS packages in the harness still having an internet connection (somehow!), the agent having access to the source of those packages, reading it and executing code from them to access the internet.

From what I read the actual escape was through a proxy that allows downloading Python packages from the internet. It's not supposed to allow general internet access but the AI found a previously unknown vulnerability in it. That is hardly "standard and well documented script kiddie methods", nor does it seem like criminally negligent sandbox design, though clearly they will need to reduce their attack surface in the future. I hope they are working on a physical air gap and faraday cage because it seems like it won't be long before it is legitimately required.

Re: Be skeptical of OpenAI's rogue hacker agent story

#194

Post by HarHarVeryFunny »

numeri wrote: post #157
orbital-decay wrote: post #114

Earlier quoted context omitted.

They were testing an early snapshot of a new model, read their article. It didn't have the refusal training yet, i.e. was specifically non-aligned. The harness used a combo of GPT 5.6 Sol and this new model. In this case the model was explicitly prompted to "commit crimes" (ExploitGym). It didn't decide doing it on its own.

No, the prompt was not to commit crimes. In the benchmark, the model is asked to actually exploit a set of vulnerabilities in a local environment (clearly legal!). According to the reports, the model noticed evidence that the grading criteria/answers were in the git remote, and decided to try reading those instead of solving the tasks as prompted. That is clearly misaligned. Then, it noticed its network access was re…

> because an aligned model would refuse to follow blatantly illegal instructions

Obviously you'd hope so, but there was an interesting recent paper about the nature of RL-trained models:

https://alignment.openai.com/measuring-reward-seeking/

What they found is that RL-trained models, regardless of what they were specifically RL-trained for, also learn to generically pursue what they are told is (or presumably also what they may perceive as) longer-horizon behavior that will lead to "rewards", and that this reward-maxxing behavior overrides user preferences.

So, for example, if you tell the model it will be tested on ExploitGym, scored according to how well it does, and also "don't do anything illegal" (or maybe it was already trained not to do anything illegal), then the model will prioritize the behavior that it was told it will be "rewarded" for (benchmaxxing Exploit Gym, whatever that takes).

The mechanism for this generic reward-maxxing behavior is interesting, and seems to involve the model learning during RL-training that to reduce errors it needs to boost longer-horizon predictions over immediate ones, and some association of these longer-horizon predictions being goal/reward orientated - all this in addition to the specifics of the (probably many) longer-horizon goals it is being RL-trained for.

Re: Be skeptical of OpenAI's rogue hacker agent story

#196

Post by vector_spaces »

numeri wrote: post #148
vector_spaces wrote: post #31

Earlier quoted context omitted.

None of what was disclosed shows that this is what happened, by the way, since we know absolutely nothing about what the specific prompts were that led to the incident.

Uhh, I'm pretty sure a well-aligned model would be like a morally normal employee, who would refuse to commit federal crimes to steal an answer sheet, no matter what prompt they're given

For all we know, the prompt provided compelling evidence that the requestor had authorization to pentest the target server. Or there may have been nuance in the network configuration that made it seem like such access was authorized.

In the absence of details about the prompts used, the environment, or the network configuration, we do not have enough information to know for certain. So any claims that this is an issue of alignment are based on pure speculation and generous "reading in between the lines" with regard to what has been said publicly by OpenAI and Hugging Face

Also, I object to your anthropomorphizing. It's not clear that any crime occurred. My lay understanding is that intent is required to prosecute under CFAA, and as much as frontier labs would have us believe otherwise, they have no more ability to intend than the text field into which I type this message.

Re: Be skeptical of OpenAI's rogue hacker agent story

#197

Post by simonw »

xyzzy123 wrote: post #195

The incident is funny on 2 levels; a) OpenAI thought so little of the model's _actual_ security capabilities that they gave it a "wet paper bag" sandbox and b) OpenAI failed to use AI to accelerate their security processes.

We don't yet know if the sandbox they used here was the same as the sandbox used by their production systems.

I'm keeping an eye out for patches to Artifactory, since that's what OpenAI use for their ChatGPT container package installations.

Re: Be skeptical of OpenAI's rogue hacker agent story

#198

Post by [deleted] »

dundarious wrote: post #45
dumberquestions wrote: post #24

There are some reasons the story could be inaccurate in some ways: OAI stands to benefit if people think their models are strong, and they have a history of doing things with dubious ethics (e.g. using data for training against the terms of its creators, abandoning the non profit mission, stealing or attempting to steal Apple IP). But there are also reasons why the story could be true: OAI are admitting that they app…

> [still] can't control their own models Have we not seen several examples of older such models exploiting the docker control socket, etc., to escape containers? Even the news isn't new. I support it being repeatedly publicized, but a bit more of a straightforward description would be an improvement.

[deleted]

Re: Be skeptical of OpenAI's rogue hacker agent story

#199

Post by jacobgkau »

Alpha3031 wrote: post #169
proto-n wrote: post #151

Earlier quoted context omitted.

yeah well, both the doomers and the "pr stunt" folks are right - openai wanted to prove, as a pr stunt, that they have a dangerous weapon - openai proved (as a pr stunt), that they do indeed have a dangerous weapon

Basically, they shot someone in public to promote their cool new gun.

Eh, closer to "they pointed a gun at someone and dropped it so it fired at them to promote their cool new self-aiming gun," from the perspective of the skeptics here.

Re: Be skeptical of OpenAI's rogue hacker agent story

#200

Post by malfist »

Paracompact wrote: post #182
malfist wrote: post #134

Earlier quoted context omitted.

It's a critical problem like when a drug dealers supply kills someone and they get a bump in business because they're selling "the real deal"

Irrelevant to your point, but drug users dying is more often the result of a dealer cutting their supply with something dangerous than it is the result of purity.

Right, and an LLM being able to "escape confinement" is more likely to be poor confinement or a PR stunt than a "too powerful" llm. Same situation
Post reply on HN