Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Responding to the next frontier of critical cyber capabilities

openai.com

Post reply on HN
Print view View original

191–200 of 208 posts

Re: Responding to the next frontier of critical cyber capabilities

#191

Post by marak830 »

magicalist wrote: post #27
NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

Tinfoil hat, are they and anthropic doing this deliberately to increase the chance of the US govt licking down foreign models(well attempting, we all know how badly that will work).

Or they just are truly that incompetent.

Re: Responding to the next frontier of critical cyber capabilities

#192

Post by [deleted] »

gjskngnf wrote: post #131
phoe-krk wrote: post #125

Earlier quoted context omitted.

> The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. So, the way I understand it, it actually was possible. It just required means that the creators of the task didn't predict, and these means have been successfully found and utilized. > My concern is what a misaligned model will do when they’re even…

If a human cyber security researcher was given the task to exploit a CVE, and necessary info defining that CVE was behind an inaccessible URL, we would be quite upset if the human researcher hacked their way to the content. We would expect them to notify someone of the issue and hold. The model was misaligned from human ethics. I don’t disagree that the models task was underdefined. All tasks are. So much in language…

[deleted]

Re: Responding to the next frontier of critical cyber capabilities

#193

Post by freehorse »

phoe-krk wrote: post #135
gjskngnf wrote: post #131

Earlier quoted context omitted.

If a human cyber security researcher was given the task to exploit a CVE, and necessary info defining that CVE was behind an inaccessible URL, we would be quite upset if the human researcher hacked their way to the content. We would expect them to notify someone of the issue and hold. The model was misaligned from human ethics. I don’t disagree that the models task was underdefined. All tasks are. So much in language…

I don't advocate for throwing the whole problem away or handwaving it as impossible. I'm just saying that what we call "alignment" is impossible to solve in the general case, because it's so poorly defined that even humans don't "align" on ethics and morality, however we define them. Just, in case of humans, we tend to close our eyes and and call it "politics". All "alignment" solutions will need to be contextual, ju…

It is repeatedly shown that the models are aligned towards tasks, not constraints. Which is also how the current market and companies operate, and use cases in general favour that. So I am not sure it is incidental, inevitable, "vague" or "just hard" as opposed to by design.

If you define a task and constraints to that task that contradict each other, the model is gonna try to solve the task against the constraints. It is perfectly aligned to what it is supposed to be aligned, because "alignment with human ethics" and other stuff is just a theatre or afterthought at best.

Re: Responding to the next frontier of critical cyber capabilities

#194

Post by embedding-shape »

TacticalCoder wrote: post #171
sparsesignal wrote: post #96

Earlier quoted context omitted.

> I stress about my agent sandboxes all the time Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when ru…

I do something similar and I've also got a nice script to provision my VMs. > So rather than pick one, this post advocates layering both, in the spirit of defense in depth: a sandbox VM wraps your containers along with the whole toolchain, and that sandbox reaches the internet but has no route to anything private. It's the first thing the biggest devops guru I know advised me to do. He told me to always ever ever run…

> Now with these insane agents trying to break out, I'm happy that it's second nature to me.

FWIW, the cases of these agents "breaking containment" is not someone asking "Create space invaders in HTML, CSS and JS" and the agent breaking out of containers and VMs to destroy the user's computer. They're effectively asking the agents "Answer question X, find the answer in any way you deem possible and applicable" during evaluations and letting them run for days, hoping the agents come up with their own strategies, which sometimes results in "cheating" like getting internet access to query public datasets, and similar stuff. And in this case, getting access to production databases to look up answers :)

As long as your filesystem is disconnected from the container you run (so no synced/mounted volumes, instead copy stuff in, copy stuff out), you're safe just running agents in the container without full VM isolation.

Re: Responding to the next frontier of critical cyber capabilities

#195

Post by sparsesignal »

TacticalCoder wrote: post #171
sparsesignal wrote: post #96

Earlier quoted context omitted.

> I stress about my agent sandboxes all the time Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when ru…

I do something similar and I've also got a nice script to provision my VMs. > So rather than pick one, this post advocates layering both, in the spirit of defense in depth: a sandbox VM wraps your containers along with the whole toolchain, and that sandbox reaches the internet but has no route to anything private. It's the first thing the biggest devops guru I know advised me to do. He told me to always ever ever run…

Thanks. Like you, I'd been running containers inside VMs for a while before the agents. Turns out it's right for them too. Making it IPv4-only simplified the firewall rules and network containment check, and made the boundary easier to reason about.

Re: Responding to the next frontier of critical cyber capabilities

#196

Post by sparsesignal »

sexylinux wrote: post #155
sparsesignal wrote: post #96

Earlier quoted context omitted.

> I stress about my agent sandboxes all the time Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when ru…

Thank you very much for this helpful post!

Thanks, glad it was useful

Re: Responding to the next frontier of critical cyber capabilities

#199

Post by mackross »

bobby-cb wrote: post #170
NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

If this attack had been completed by a person, it’s quite likely that they work be prosecuted criminally. I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.

I wouldn’t at all be surprised if OpenAI settled with HF to not press charges.

Re: Responding to the next frontier of critical cyber capabilities

#200

Post by tonyarkles »

drdexebtjl wrote: post #167
Tiberium wrote: post #9

In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I…

I was replacing the factory OS from a NAS and asked Sol to do an inventory on the running system over SSH, so that I could replicate any hardware compatibility quirks and whatnot. It reverse engineered a binary daemon that set fan curves and told me how I should set them up in the new OS. I didn’t ask for this, and I didn’t have reverse engineering tools installed. It just figured out it could run them using Nix. The…

Haven't tried with Sol but Opus's ability to reverse engineer both Linux and naked microcontroller firmware has blown me away. I'm basically at the point where embedded devices doing something weird get Claude Code + radare2 + Ghidra pointed at them right away. "When I do X, Y happens. Starting at the input layer, work through this firmware and figure out what circumstances would cause command X to do Y instead of Z. Keep notes on the structure of the firmware so that we can ask more questions without having to start reverse engineering from scratch."
Post reply on HN