Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Responding to the next frontier of critical cyber capabilities

openai.com

Post reply on HN
Print view View original

171–180 of 208 posts

Re: Responding to the next frontier of critical cyber capabilities

#171

Post by TacticalCoder »

sparsesignal wrote: post #96
magicalist wrote: post #27

Earlier quoted context omitted.

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

> I stress about my agent sandboxes all the time Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when ru…

I do something similar and I've also got a nice script to provision my VMs.

> So rather than pick one, this post advocates layering both, in the spirit of defense in depth: a sandbox VM wraps your containers along with the whole toolchain, and that sandbox reaches the internet but has no route to anything private.

It's the first thing the biggest devops guru I know advised me to do. He told me to always ever ever run my containers inside VMs. I religiously followed his advice ever since and I couldn't be happier: I was already doing it before the AI days, to run "normal" containers. Now with these insane agents trying to break out, I'm happy that it's second nature to me.

It's funny that, just like you, I'm using IPv4-only too for those VMs.

I tried GPU-passthrough as a proof-of-concept (worked fine) but I've got no use for it yet.

FWIW all my VMs are running on a 10 years old Xeon / 14 cores / 28 threads / ECC RAM. That's where the agents live.

Thanks for the link to your write-up, very interesting to read from like-minded people and see what's similar and what's different in their solutions.

Re: Responding to the next frontier of critical cyber capabilities

#172

Post by TacticalCoder »

Tiberium wrote: post #9

In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I…

> Although I think most vulnerabilities are going to be closed in popular software by mid 2027, ...

If we're to believe how good those AI are at CTF and at escapes of all kind, then the only logical conclusion is that, by very far, most vulnerabilities were already closed, even before AI.

Otherwise we'd already be in deep shit since months if not years.

Re: Responding to the next frontier of critical cyber capabilities

#173

Post by brandon272 »

bobby-cb wrote: post #170
NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

If this attack had been completed by a person, it’s quite likely that they work be prosecuted criminally. I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.

They aren’t prosecuted because they’re in a club that you are not in. Not all citizens have to abide by the same laws and rules.

Re: Responding to the next frontier of critical cyber capabilities

#174

Post by fn-mote »

kmeisthax wrote: post #114
magicalist wrote: post #27

Earlier quoted context omitted.

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

To make matters worse, they let the model train on it's own exploits?! If I were running this thing, my first thought would be to keep those reasoning traces quarantined, stop and delete the current post-training run, and start fresh AFTER they'd fixed the sandbox RCE. Don't let the model learn about the holes in its environment. We have the AI equivalent of Class D and F amnestics and we're not using them?! OK, yes,…

Amazing story.

I think you’re defending against a level of aggression that most of the world doesn’t believe exists yet.

I suppose by the time we believe, it will be too late.

There’s a lot of people who will be trying to run LLMs with hacking goals, so maybe we need to get used to a lot of automated attacks.

Re: Responding to the next frontier of critical cyber capabilities

#175

Post by jadar »

sailingparrot wrote: post #145
jadar wrote: post #68

Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?

> Isn't this the opposite of what everyone is saying should happen? I think "everyone" is doing heavy lifting here. It's not clear to me at all that a powerful model released with no restrictions would be a net positive. This hinges on the hope that the under paid, under motivated, under staffed and under qualified security teams at many random corps are going to leverage those open models to fix their vulns faster (…

You're right. But I used it specifically because of the open letter calling for open models — or at least for not banning them — which went around recently and represented a very large number of tech and AI companies. Anthropic seemed the only exception.

Re: Responding to the next frontier of critical cyber capabilities

#176

Post by janalsncm »

jackb4040 wrote: post #166
janalsncm wrote: post #156

Earlier quoted context omitted.

> Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power They aren’t just an “AI company”. They’re the primary entry point of open weight models. If open weight models are seen as dangerous as a result of this incident, it will be bad for them. Similar to how it would be bad GitHub if open source was seen as dangerous.

Can you concretely lay out how you think that scenario would negatively impact HF? They're the de-facto central distribution point for all models; it seems overwhelmingly likely that increased censorship benefits them relative to any competitors. It's hard to imagine their internal culture is anything other than saturated with AI fanboys whose reaction to OpenAI hacking them was to point out how it reminded them of t…

An open weight model is used in a cyber attack. Responsibility is hard to attribute. The US/EU/any other government sees them as the distributor of the open weight model and comes after them. Passing a law making HF liable for any damage caused by a model they host.

Re: Responding to the next frontier of critical cyber capabilities

#177

Post by xur17 »

porridgeraisin wrote: post #120
user43928 wrote: post #76

Earlier quoted context omitted.

I thought you need to prove you are working in cybersecurity or provide evidence of authorization for work done. It's really just simple ID/face verification?

There is some additional check, not sure what it entails, some people I know and I passed it, others failed.

Interesting.. I just went through the process and passed KYC, but was told my OpenAI that I was rejected from the program. Guessing some device check rejected me.

Re: Responding to the next frontier of critical cyber capabilities

#178

Post by breadramen »

bottlepalm wrote: post #152
somesortofthing wrote: post #129

I'm not convinced that there's any amount of monkey-patching you to fix the problem of "we now have AI that actively needs strong containment measures lest it start coordinating in secret with other instances to do real-world damage."

In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.

[dead]

Re: Responding to the next frontier of critical cyber capabilities

#179

Post by noman-land »

bobby-cb wrote: post #170
NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

If this attack had been completed by a person, it’s quite likely that they work be prosecuted criminally. I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.

If I left a script running for 4 days that hacked multiple companies the FBI would be at my house with 15 people and 35 guns. These guys get to do presentations about it.

Re: Responding to the next frontier of critical cyber capabilities

#180

Post by [deleted] »

Tiberium wrote: post #9

In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I…

[deleted]
Post reply on HN