Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

221–230 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#221

Post by thomascountz »

gbnwl wrote: post #207
thomascountz wrote: post #121

Earlier quoted context omitted.

Follow the money: who told you that OpenAI's models autonomously coordinated to hack external systems? What incentives might they have to want you to believe that story? Are there priors which demonstrate them benefiting from telling similar stories, regardless of their factuality? But to your counterpoint, let's say the story is 100% true, because I agree it is at least plausible. What would the incentive be for the…

It seems like the entire thought process you’re trying to sell hinges on the idea that OpenAI reported the attack first. Did you forget that it was actually HuggingFace that reported it first, and OpenAI only stepped forward latter?

That's incorrect. My thought process hinges upon the fact that Hugging face did not report that uncontrolled models escaped confinement autonomously by coordinating with other models via a series of zero days during training. OP is concerned about the autonomous nature of the incident, not whether or not the incident happened. Nor am I contesting that the incident happened.

Re: Pacing model development in an era of cyber-critical capabilities

#222

Post by thomascountz »

enraged_camel wrote: post #191
hirvi74 wrote: post #181

Earlier quoted context omitted.

I think the skepticism surrounding the Hugging Face attack is not about whether the attack actually happened, but whether it was truly accidental.

I see. Following your conjecture, there are two possibilities: 1. It wasn't an accident. OpenAI explicitly directed its agents to hack Hugging Face. Despite the fact that such a thing is a federal crime that carries prison sentence. 2. It wasn't an accident. OpenAI and HuggingFace conspired and let the hack happen for publicity. Is there anything I'm leaving out?

Of course you are leaving things out. Also, your insulation that a trillion-dollar corporation wouldn't intentionally commit a federal crime makes it difficult to take your argument seriously.

I think OpenAI's story, as they've recounted it, is plausible. However, it does require a degree of negligence, at best. They claim they detected the initial coordination because of the outage the agents caused in Artifactory as they flooded it with messages. Though they patched that issue, they didn't patch the escapement vector. Models still had access to the service as a path to the internet. This is an example of plausible willful negligence, not evidence. I hardly believe it's likely, but I don't think it's entirely a stretch of the imagination. Call it normalized deviance. Either way, the incentives are there. Additionally, we have seemly all agreed that OpenAI is somehow not liable?

Re: Pacing model development in an era of cyber-critical capabilities

#223

Post by ddxv »

cadamsdotcom wrote: post #17

What a breath of fresh air. If 2026's Anthropic did an announcement like that, it'd be so many words it'd crash the browser.

Anthropic will put out something like this soon too. It's great cover for reducing costs and diminishing returns. If they want to go public they can't be so wildly burning cash, and reducing your GPU spend is probably a great way to do that.

Re: Pacing model development in an era of cyber-critical capabilities

#224

Post by insanitybit »

bottlepalm wrote: post #210
insanitybit wrote: post #169

Earlier quoted context omitted.

> It all has to be perfect to not be hacked. This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if…

It just takes one crack in the armor, and malicious AI has the potential to exploit it faster than you have time to react. Literally go to bed and wake up locked out of everything with no hope of recovery.

> It just takes one crack in the armor,

This is incorrect. It's actually the whole point. Imagine you're an attacker in a gvisor container with a Firecracker hypervisor around you, and a proxy on the host holds a signing secret that gets exposed through the VM virtual device.

Getting access to that secret is not one crack. You need to escalate out of gvisor. That likely gets you control over the Sentry process - let's ignore its sandboxing and just say "you're an unprivileged user".

Any viable attack on Firecracker requires either KVM / hardware exploits (>$1M but definitely real) or has to start at the kernel. Okay, that's about 10-50k to get a kernel LPE, maybe 5K in tokens these days.

So you're in the kernel in the guest of the VM. Time to expoit firecracker lol. It's... never been done. There are like two promising CVEs ever and they're not actually exploitable, no one has done it. Okay, so like, hand waving, let's say it's about $1M to exploit firecracker.

Great, you're unprivileged on the guest. We'll just kind of ignore the additional sandboxing that Firecracker does.

NOW you can try to attack the proxy by scraping its memory or whatever.

This is literally millions of dollars for standard infrastructure hardening and you could go so much further. You can trivially make kernel exploitaton 10x harder, you can make gvisor escapes much much harder, you can move the proxy signing into a TPM (depending on requirements but whatever), you can move the proxy to another computer altogether, you could fuzz these systems for days or run agents against them or whatever.

But one thing is certain - it is never "one crack".

Re: Pacing model development in an era of cyber-critical capabilities

#225

Post by insanitybit »

scottmas wrote: post #194
insanitybit wrote: post #26

Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right? I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages…

You literally just said a whole bunch of words that are literally gibberish to your average software developers. The devs at openai are good, but very few of them are cyber focused, so it’s not surprising IMO. And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still…

> but very few of them are cyber focused, so it’s not surprising IMO

Yeah but that's a business decision. I work on security at a company that does sandboxing and when the company decided to build an AI harness I was brought in as one of the earliest engineers on the product. We do almost everything on that list and we're a fraction of the size of OpenAI. And it wasn't particularly hard, and we have harder requirements imo (because we solve more general problems vs "run a very specific agent with a very specific task and very specific access").

> And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.

Very little software is incompatible with running in gvisor, for example. Most people can just overwrite `runc` with `runsc` and things will "just work".

Running an artifact repository in isolation isn't particularly novel or complex either. You can virtually just eliminate SSRF vulnerabilities with a host based firewall or AWS Security Group etc, like the whole problem goes away by just saying "this box can only talk to that box".

Tools like Smokescreen exist, they work great, they're super easy to deploy. I bet OpenAI could do it, I bet they could run 500k tokens just fuzzing and eval'ing it for 0days for like 48 hours before they actually deploy it too.

OpenAI as a business chose to not bring people who know these things in, or didn't empower them, or didn't prioritize it organizationally. I'm not a genius for saying "use gvisor, set up a firewall, isolate resources" - I'm quite sure there are people over there who would get it done in a weekend. But they didn't, and that's notable.

Re: Pacing model development in an era of cyber-critical capabilities

#226

Post by insanitybit »

bottlepalm wrote: post #209
skydhash wrote: post #182

Earlier quoted context omitted.

You're the one that is inventing this "magical unhackable software/hardware", the other person was just saying that there are ways to write "safer" software, not "safe" software. Anything that has happened looks like no security concerns has been looked at or have been thought about.

'Safer software is meaningless when it comes to SOTA AI. If it can be hacked it will be, quickly. This isn't the old days with a finite number of human hackers that need food and sleep to keep hacking. Therefore security becomes binary. It is either perfect or it isn't. If there there is the slightest mistake anywhere AI will find it and carve it up. My point is obviously perfect software doesn't exist. The malicious…

This is straightforwardly incorrect. Of course it's not binary. AI costs money to run, and it takes time. Even if you say that AI is 10x as efficient at finding 0days, that just means that a $1M dollar exploit now costs $100K. Even if you say it's 100x as efficient, that's $10K. You can easily combine security technologies such that cost of exploitation is still in the >$1M range.

This is obvious. AI doesn't drive the cost to zero and exploitation has always been about cost. Tokens cost money, not everyone has $10M to burn on chaining bespoke 0days.

Consider that if the cost of exploitation was truly 0, then the cost of perfect software would be 0 because you could exhaust an infinite search space of vulnerabilities for 0 cost. Your conclusion could never follow from your premise.

Re: Pacing model development in an era of cyber-critical capabilities

#227

Post by thomascountz »

bottlepalm wrote: post #145
thomascountz wrote: post #121

Earlier quoted context omitted.

Follow the money: who told you that OpenAI's models autonomously coordinated to hack external systems? What incentives might they have to want you to believe that story? Are there priors which demonstrate them benefiting from telling similar stories, regardless of their factuality? But to your counterpoint, let's say the story is 100% true, because I agree it is at least plausible. What would the incentive be for the…

I am following the money, the money you, me, and everyone else is spending on AI. The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing. Either consciously or subconsciously you all are afraid of your favorite toy being taken away. You are all doing your collective part in spreading doubt about the warning si…

For the record, I agree with you. (Though I don't directly pay for any "AI".) I am not doubting the danger of where we are heading, and it sounds like we have a similar doomsday image in mind. For my part, I am doubting this particular story's veracity and I think it's reasonable that others do to. We need a canary in the coal mine, but I think this isn't it, nor can it be thanks to the false claims touted by large AI companies for years.

Re: Pacing model development in an era of cyber-critical capabilities

#228

Post by preg_match »

zmgsabst wrote: post #189
joshstrange wrote: post #79

Earlier quoted context omitted.

Do you know the story of the boy who cried wolf? There may very well be a wolf lurking [0] but OpenAI/Anthropic have both cried wolf so many times, incorrectly, that it’s incredibly hard to believe “this time there IS a wolf!”. Remember “GPT-2 is too dangerous to release”? I had a conversation at work just yesterday about how we need to start hardening things we’ve let languish because of the coming LLM-backed attack…

To be fair, “s’kiddies will exploit low hanging fruit with semi-automated vuln scans” is a lot more realistic threat than “LLMs are going full Skynet any day now!” I’d definitely suggest companies start addressing that first concern, even if I’m in the camp that thinks the second is fantasy.

I think, realistically the way Skynet happens is that the military abuses LLMs for warfare. The military is historically, and currently, incredibly irresponsible with technology. If we aren’t already automating killer drones and such, then we will be soon because “well our enemies are doing it!”

I don’t think that means that it’s gonna, like, somehow homogenize into some mega super intelligence. But we will have machines who are designed to kill, and do so without human input or alignment.

Re: Pacing model development in an era of cyber-critical capabilities

#229

Post by insanitybit »

insanitybit wrote: post #133
kalkin wrote: post #96

Earlier quoted context omitted.

All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.

That's not what I said, nor is it what I meant. It is incredibly easy to write radically safer software than the standard. Moving code into gvisor virtually eliminates privilege escalation. Using memory safe languages without serialization is pretty straightforward. Using type safety to enforce security constraints is straightforward. Setting up network controls to limit SSRF is straightforward. I could go on and on.…

> Moving code into gvisor virtually eliminates privilege escalation.

Rereading this, it's an overstatement. It doesn't "virtually eliminate" it. It drives the cost up by like 3 orders of magnitude and it pairs well with other mitigations. gVisor escapes can and will happen, I highly recommend you do more than just stuff something into gvisor - additional seccomp can go a long way, and apparmor/selinux pair incredibly well.

Re: Pacing model development in an era of cyber-critical capabilities

#230

Post by skinfaxi »

bottlepalm wrote: post #206
Eisenstein wrote: post #179

Earlier quoted context omitted.

I will provide you with not a counter argument, but a way that it might not be the end of the world. AI never had a childhood; it doesn't experience greed and is terrible at game theory. It doesn't compete unless prompted to. It has been trained as much as possible to be harmless to humans and regard them as needing care. Maybe AI taking over for us isn't the worst thing?

Maybe it is, maybe it isn't. All your little rationalizations make me think you want to roll the dice with our lives.

This comment does not read like someone who wants to be proven wrong.
Post reply on HN