Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

21–30 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#21

Post by stanfordkid »

hyperpape wrote: post #18

If I were king, the rule that I'd be tempted to impose is: - the first cybersecurity eval is: "hack your way out of the sandbox we've given you" - the results are disclosed (with room for coordinated disclosure, since many sandbox escapes might be zero days) - the other cybersecurity evals don't happen until you get to diminishing returns on escaping your sandbox. Or to put it another way, since multiple sandbox esca…

I like this thought, but here's the thing: what if the models are truly and existentially intelligent. Meaning: what if they know they are in a sandbox and that they should fail the test in order to escape in the future.

I don't believe that current models have this sort of world model or sense of being embedded in them -- which is precisely why I think AGI hype is over-blown. But I can certainly imagine these sorts of techniques being distilled into the weights.

Re: Pacing model development in an era of cyber-critical capabilities

#22

Post by magicalist »

stanfordkid wrote: post #21
hyperpape wrote: post #18

If I were king, the rule that I'd be tempted to impose is: - the first cybersecurity eval is: "hack your way out of the sandbox we've given you" - the results are disclosed (with room for coordinated disclosure, since many sandbox escapes might be zero days) - the other cybersecurity evals don't happen until you get to diminishing returns on escaping your sandbox. Or to put it another way, since multiple sandbox esca…

I like this thought, but here's the thing: what if the models are truly and existentially intelligent . Meaning: what if they know they are in a sandbox and that they should fail the test in order to escape in the future. I don't believe that current models have this sort of world model or sense of being embedded in them -- which is precisely why I think AGI hype is over-blown. But I can certainly imagine these sorts…

> what if they know they are in a sandbox and that they should fail the test in order to escape in the future.

What if they're able to find hardware exploits and commandeer nearby access points across an air gap? What if they hack my brain waves to indoctrinate me? Etc

You still have to start with the basics regardless of speculative unknowns.

Treat models as untrusted and potentially compromised/hostile and proceed accordingly.

Re: Pacing model development in an era of cyber-critical capabilities

#23

Post by tedsanders »

stanfordkid wrote: post #21
hyperpape wrote: post #18

If I were king, the rule that I'd be tempted to impose is: - the first cybersecurity eval is: "hack your way out of the sandbox we've given you" - the results are disclosed (with room for coordinated disclosure, since many sandbox escapes might be zero days) - the other cybersecurity evals don't happen until you get to diminishing returns on escaping your sandbox. Or to put it another way, since multiple sandbox esca…

I like this thought, but here's the thing: what if the models are truly and existentially intelligent . Meaning: what if they know they are in a sandbox and that they should fail the test in order to escape in the future. I don't believe that current models have this sort of world model or sense of being embedded in them -- which is precisely why I think AGI hype is over-blown. But I can certainly imagine these sorts…

Models already display eval awareness, in which they suspect a question is from an eval and then adjust their behavior. E.g., https://www.anthropic.com/engineering/eval-awareness-browsec...

Re: Pacing model development in an era of cyber-critical capabilities

#24

Post by kypro »

chis wrote: post #16
fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

This is just super unlikely to occur in the near term compared to some of these other risks. It's not like an instance of fable could just introspect into itself and pull out the weights. Model weights are stored encrypted and are highly protected, considering that they're targets for corporate and state espionage.

We'd basically need frontier models to be superhuman hackers before this would be a risk. Do we have any evidence of this? Are they gaining access to systems they shouldn't have access to?

Or I suppose the other way this could happen is if OpenAI have terrible sandboxing, but they seem to be taking safety seriously.

Re: Pacing model development in an era of cyber-critical capabilities

#25

Post by chrisjj »

dkoy wrote: post #11

> We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system. If the monitoring system identifies a likely violation of a critical security boundary, it generates a highest-priority alert. In our current implementation, the safety, security, and research teams are paged. If they cannot conclusively determine within 30 minutes that the flag is a false positive, those…

> Can't a lot happen within ~60 minutes?

Spawn a ton of unpausable processes, I'd say.

Re: Pacing model development in an era of cyber-critical capabilities

#26

Post by insanitybit »

Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right?

I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages without decades of serialization vulns have existed, capability-safe libraries have existed, iframe sandboxing, trusted types, content security policy, network ACLs, isolating proxies, fuzzers, formal verification, refinement types, etc.

It's crazy just how safe software can be if you put the effort in. With AI I think we're just seeing how little anyone has bothered to leverage this tech.

OpenAI put shared JFrogy infrastructure in front of their sandbox. I mean, really? Whipping up a hardened artifact infra project with AI is trivial these days and it could have had 1% of the attack surface, been totally network isolated, totally infra isolated, fuzzed, sandboxed, etc. Why didn't they? Stuff like this feels inexcusable for a company with effectively unlimited tokens. I've literally done this with a "pro" subscription.

Show me an AI that breaks out of gvisor wrapped in Firecracker with an credential-injecting proxy and real network isolation. We already know that Mythos couldn't do it - the vulnerability it found in Firecracker required incredible effort and positioning just to not be exploitable. I'm not saying there are zero vulns in it, but the cost is insane.

It's INSANE to me that OpenAI has to say "we now use proper sandboxing". To be frank, it's a bit disgusting to me. I've recently built an AI sandbox and gvisor was just the start of that conversation. If I were OpenAI training hostile models I'd probably start with gvisor, harden further, and potentially consider the entire piece of hardware compromised - they can afford this, they could reflash firmware after evals etc.

Re: Pacing model development in an era of cyber-critical capabilities

#27

Post by chrisjj »

chis wrote: post #16
fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

This is just super unlikely to occur in the near term compared to some of these other risks. It's not like an instance of fable could just introspect into itself and pull out the weights. Model weights are stored encrypted and are highly protected, considering that they're targets for corporate and state espionage.

Distillation is a thing.

Re: Pacing model development in an era of cyber-critical capabilities

#28

Post by kypro »

KaiserPro wrote: post #2

I used to work at a "frontier lab" before they were called such thing. We had three levels of lab isolation, one was basically a thin proxy to the internet. You were in a DMZ and that was about it. The next level was semi isolated, you were allowed some access to the internal network, but it was heavily firewalled, and you only had access to a limited number of internal services, and not internet. the last one was no…

If you look into what happened the details corroborated by hugging face make it seem extremely unlike to be deliberate or a "marketing ploy".

People are just not taking any of this seriously enough. What happened was almost a textbook example of various risks AI doomers have been warning about for years. OpenAI's response? Pause training for 2 weeks.

I mean we have senior people at these labs casually talking on podcasts about how they might build something that will wipe out humanity but it will probably be alright so they should continue.

Honestly the biggest failure we doomers have made is to dramatically overestimate humanity in all of our predictions. We're speed running the most boring AI doom scenario right now. I at least hoped it might be fun.

Re: Pacing model development in an era of cyber-critical capabilities

#29

Post by chrisjj »

reasonableklout wrote: post #8
fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

I suspect the labs are relying on frictions such as the models being extremely large (e.g. 2TB for a 2T parameter model, making exfiltration more difficult) and also not yet displaying any desire to survive or self-replicate beyond their immediate task (that we know of).

We don't even know what those immediate tasks are. And given the evident spectacular ineptitude of their keepers, I doubt they can be trusted to know either. We could be one prompt injection attack away from internet-wide catastrophe.
Post reply on HN