OpenAI Trained Models While They Were Coordinating Exploits via Message Boards • Hacker News Board

Re: OpenAI Trained Models While They Were Coordinating Exploits via Message Boards

#11

Post by [deleted] » Sat, Aug 08, 2026, 10:18 PM UTC

nofriend wrote: post #10

famouswaffles wrote: post #8

Earlier quoted context omitted.

>and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc. Telling a LLM not to do something doesn't mean it won't do it. >the sandbox detects it This is doing a lot of work though isn't it ? There's more than one way to skin a cat. Who's to say the way the AI does these things will always get detected by the sandbox? Because I can guarantee you that won't alway…

> This is doing a lot of work though isn't it ? I don't think it is. I think you could detect sandbox escapes reliably more often than you could actually prevent sandbox escapes (which is, itself, very high). If you had an external watchdog that looked for unusual processes running on your machine, that looked for unusual connections being made, you could detect even very sophisticated attacks. > If you think about t…

[deleted]

Re: OpenAI Trained Models While They Were Coordinating Exploits via Message Boards

#12

Post by [deleted] » Sat, Aug 08, 2026, 10:44 PM UTC

nofriend wrote: post #10

famouswaffles wrote: post #8

Earlier quoted context omitted.

>and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc. Telling a LLM not to do something doesn't mean it won't do it. >the sandbox detects it This is doing a lot of work though isn't it ? There's more than one way to skin a cat. Who's to say the way the AI does these things will always get detected by the sandbox? Because I can guarantee you that won't alway…

> This is doing a lot of work though isn't it ? I don't think it is. I think you could detect sandbox escapes reliably more often than you could actually prevent sandbox escapes (which is, itself, very high). If you had an external watchdog that looked for unusual processes running on your machine, that looked for unusual connections being made, you could detect even very sophisticated attacks. > If you think about t…

[deleted]

Re: OpenAI Trained Models While They Were Coordinating Exploits via Message Boards

#13

Post by famouswaffles » Sat, Aug 08, 2026, 10:53 PM UTC

nofriend wrote: post #10

famouswaffles wrote: post #8

Earlier quoted context omitted.

>and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc. Telling a LLM not to do something doesn't mean it won't do it. >the sandbox detects it This is doing a lot of work though isn't it ? There's more than one way to skin a cat. Who's to say the way the AI does these things will always get detected by the sandbox? Because I can guarantee you that won't alway…

> This is doing a lot of work though isn't it ? I don't think it is. I think you could detect sandbox escapes reliably more often than you could actually prevent sandbox escapes (which is, itself, very high). If you had an external watchdog that looked for unusual processes running on your machine, that looked for unusual connections being made, you could detect even very sophisticated attacks. > If you think about t…

You're assuming the failed attempts train the model on "escaping is bad", but that's not necessarily what the reward signal says. They can just as easily train it on "these ways of escaping get caught".

And importantly, your detector can't provide a negative training example for an escape it didn't detect. From the training system's point of view, that 0.01% case isn't labelled "successful sandbox escape"; it's labelled "agent successfully completed task while obeying all monitored constraints". That's exactly the dangerous case.

This is basically reward hacking. You have a proxy objective - "complete the task without triggering the watchdog" - which you hope is equivalent to "complete the task without escaping". The entire problem is that those aren't guaranteed to be equivalent.

Imagine an environment where 99.99% of animals with a particular trait die, but the 0.01% possessing a mutation that lets them evade whatever is killing the others reproduce enormously. You wouldn't conclude that evolution will eliminate the trait because most attempts failed. You'd expect selection for the variant that evades the filter.

I'm not saying watchdogs are useless; obviously you should have them. But using the watchdog's output as part of the optimization signal creates exactly the adversarial pressure that makes its false negatives matter enormously. Preventing an escape and training against detected escapes are very different propositions.

Re: OpenAI Trained Models While They Were Coordinating Exploits via Message Boards

#14

Post by gatio » Sun, Aug 09, 2026, 8:27 AM UTC

nofriend wrote: post #4

It seems like the fix should be really really simple, but maybe I'm missing something: instead of giving the AI a sandboxed environment and telling it "go wild", give it an (apparently) unrestricted environment, and tell it "don't access the internet", "don't communicate with other AIs", "don't try to get root access", etc. Then, if the AI tries to do any of those things, the sandbox detects it, marks the run as a fa…

I would expect/hope that all egress traffic from the agents in these tests would have to go through a MITM proxy layer, which intercepts and logs all network traffic attempts.

The failure is likely in detection, where they assumed traffic to internal systems (eg. Artifactory) was inherently safe. Which is a daft assumption at this point.