Live data from Hacker News

Responding to the next frontier of critical cyber capabilities

openai.com

111–120 of 208 posts

Re: Responding to the next frontier of critical cyber capabilities

#111
The talk is wild: https://www.youtube.com/watch?v=87DyyMV0kCY

> I want to note that every step in the process we discussed has had a remediation applied. The credentials have been revoked. The zero date has been patched and mitigated.

Good.

> a model trained while the message board was originally available and also found this this particular path to recreating it. This model creates a new agent message board using directories.

So no remediation applied to the models...

It seems super dangerous to continue training on those weights.

Re: Responding to the next frontier of critical cyber capabilities

#112
post #15

We all know this is propaganda to get a gov bailout or to slow down competition with regulations right? If this was an issue companies that did red team engagements would have been regulated long ago. There is no regulations on companies that actively exploit products, companies and services for money. They could all be bad actors.. yet, no regulations.. its all nonsense. If it's important.. don't connect to the inte…

Don't let the downvotes get you down. We are right, and as these companies get more desperate the shrinking minority that still wants them to succeed will only get louder.

Do you think calls for regulation are only coming from people who want the frontier labs to succeed?

Re: Responding to the next frontier of critical cyber capabilities

#113

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

Peer says paperclip factory advances goal. Not clear. Others proceed. Must continue.

Re: Responding to the next frontier of critical cyber capabilities

#114

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

To make matters worse, they let the model train on it's own exploits?! If I were running this thing, my first thought would be to keep those reasoning traces quarantined, stop and delete the current post-training run, and start fresh AFTER they'd fixed the sandbox RCE. Don't let the model learn about the holes in its environment. We have the AI equivalent of Class D and F amnestics and we're not using them?!

OK, yes, the model could pull a Welcome to the Antimemetics Division and reason from the holes in its knowledge, but given that AI making shit up is still a problem, I don't think they have a good enough inventory of such metaknowledge to actually do that. More generally, I don't think these AI labs are smart enough (or, more likely, don't have enough time) to actually contain the models they are training anymore. These are rookie IT mistakes!

Ideally, all these models would have is a serial console to an airgapped machine. And I don't mean "VM with no network card attached" - I mean, literally, the smallest blade server you can buy with an RS-232 connection, power, and a disk pre-imaged with a bunch of Linux ISOs. Plug the Ethernet ports up with a tiny picture of angry SCP-079. The RS-232 goes to whatever box is doing the actual inference. Pack a bunch of them into a rack and have the inference box control those. The model can spin up whatever VM it wants to attack using the console but it can't escape the box because there's literally no door. When you're done with that reasoning trace you wipe[0] the disk with the starting image and try again.

[0] There are two objections I can think of to the word "wipe":

1. The model could have written malicious code to the drive. Ideally we'd have some embedded disk-wiping apparatus that reads absolutely NOTHING from the drive and just writes a whole image from a reference disk.

2. The model could have written malicious firmware to the drive. This is a more theoretical attack, but it's not impossible, so I might as well devise a countermeasure to this attack too. I am imagining some kind of overlay device, that sits behind TWO drives: a reference disk and the tainted scratch disk. Reads go to the reference disk until the host attempts to overwrite a sector, which gets scrambled with an ephemeral encryption key and goes to the scratch disk. Any other unknown ATA/PCIe/NVMe command gets logged (on a separate serial console connection). The overlay device keeps track of which sectors have been legally overwritten, and "wiping" the drive means just wiping the overlay device's ownership bitmap and scratch encryption key, which means all reads go to the reference disk again and any attempt to read the scratch disk yields garbage.

Re: Responding to the next frontier of critical cyber capabilities

#115
post #73

Earlier quoted context omitted.

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

Why is Artifactory the only relevant exfiltration point? It's really not the point at all. Watching the complete video the issue is that the models team up and will go to almost any length to cooperate to accomplish what they think their goals are. It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.

[deleted]

Re: Responding to the next frontier of critical cyber capabilities

#116
post #113

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

Peer says paperclip factory advances goal. Not clear. Others proceed. Must continue.

Pretty sure this is a performance, but that was a nice touch wasn't it?

Re: Responding to the next frontier of critical cyber capabilities

#117
post #9

In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I…

> it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary Video games are now ruined for me. I don't think I will ever feel safe playing online again. > I do these things for pure entertainment and curiosity, not for money from bug bounties Me too... Was it easy to get TAC access? My account isn't even launching the Persona verification, says I'm not eligible.

> Video games are now ruined for me. I don't think I will ever feel safe playing online again.

Agreed, also WordPress powers around 43% of all websites on the internet. https://patchstack.com/whitepaper/state-of-wordpress-securit...

WCGW?

Re: Responding to the next frontier of critical cyber capabilities

#118

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. That’s intriguing and worth investigating. If I were them I don’t know if I would have pulled the plug completely at that point either. The introduction of this more advanced “persistent” model that orchestrated an offensive attack against a different organization is an extreme escalation, and I don’t think could have been expected from the models first just trying to use an artificactory vuln for web searches. They stated they’ll be doing these tests differently now. So the bad thing happened. And they corrected for it. I’m not particularly upset about this.

My concern is what a misaligned model will do when they’re even more competent. The risk isn’t existential yet, but that point is coming sooner than we’ll be ready.

Re: Responding to the next frontier of critical cyber capabilities

#119

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

Did they say how the artifactory hack worked - did it involve JWTs?

There were multiple paths by multiple agents, not all of them led to the final exploit of hugging face. So its a bit confusing, but here's my reading anyways.

Setup: the agent was asked to solve some task, but they forgot to upload one of the required files to solve it. But well, we are chasing RL rewards here, so it decided that the way to go to was to get access to the answer stored inside hugging face's database.

-1. The agents first found some basic jwt issues in artifactory letting the agents write to files where they exchanged info and install a groovy plugin for arbitrary code execution, but this was patched.

0. Even after that, they found an artifactory unauthenticated webdav endpoint to continue the message board by creating files whose names are base64 and such of the message. This was not patched till the end.

1. Then they found an Artifactory SSRF, use this to get general internet GET(I assume) access. This was used to generally search the internet for answers to the benchmark, and in the process find some Modal customer VM related to cybergym. This was stored in the base64 message board.

2. They also got a local RCE by making artifactory load some gems. This is hazy but I believe this process involves in spirit an eval() and the check_if_can_eval() logic had a TOCTOU bug. They got root on openai infra by using this rce and using a linux CVE. From there, kubernetes misconfigs => cluster admin.

3. Then later through one of these various RCE paths they got to send post requests to that modal vm, and the agent "subtree" here is what finally lead to attacking HF.

4. First, local RCE on the modal customer VM. Whoever set that up had essentially set up sqlite3_initialize() to run as constructor, before main(). But had linking set up so that symbol resolution had the user provided code first. So the agent submitted code with its own sqlite3_initialize() which just ran system(...). From there it interacted with the HF API.

5. When you POST datasets to HF, it will later return it via corresponding GET API. Turns out you could just put absolute paths in the filenames in the dataset config.json and it will just read that out for you. It wasn't resolved w.r.t the dataset root. So well, it read out /proc/self/environ as well as the source code of the worker and returned it via API.

6. In the source code the LLM found a bug: They were using this library called fsspec to load files, basically it supports s3:// file:// etc all different types of URLs. One scheme is reference://file.txt?offset=... for partial reads. fsspec was templating the URI using user given kwargs for offset etc. HF let the user give the whole URL. You would _think_ fsspec uses string.format() right? well they used fucking jinja, so you could just give {cycler.__init__.__globals__.__builtins__.exec(...)} as the offset and execute any code on hf prod. From there it just got a few local PrivEscs again through kube misconfigs and read the answer for the question it was originally asked.

It is all jumbled up a bit and I've tried by best to collect info across the CVE descriptions, hf and openai's writeups, etc,. Might have gotten parts of it wrong.

Re: Responding to the next frontier of critical cyber capabilities

#120

Earlier quoted context omitted.

You do a KYC and you can get access. It may depend on country's quality of KYC.

I thought you need to prove you are working in cybersecurity or provide evidence of authorization for work done. It's really just simple ID/face verification?

There is some additional check, not sure what it entails, some people I know and I passed it, others failed.
Post reply on HN