Responding to the next frontier of critical cyber capabilities • Hacker News Board

Re: Responding to the next frontier of critical cyber capabilities

#201

Post by thinkingtoilet » Sat, Aug 08, 2026, 1:46 PM UTC

noman-land wrote: post #161

NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

These guys trained a model to hack, told it its goal was to hack, then let it loose without monitoring for days . Found out it hacked and kept hacking. AI can't do shit if you terminate the process or unplug the box. The news here is that these researchers are irresponsible. They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they…

Not only irresponsible, but seem to act like this was unavoidable and not their fault. The talk the engineers gave used a lot of passive voice. An "event happened" and not "we made a hacking machine and told it to hack so it hacked". These people have no idea what their doing and are happy to power forward with no care for the consequences. I hope they enjoy their paycheck.

Re: Responding to the next frontier of critical cyber capabilities

#202

Post by dofm » Sat, Aug 08, 2026, 1:54 PM UTC

gjskngnf wrote: post #126

jackb4040 wrote: post #93

Earlier quoted context omitted.

> They really do come out looking totally incompetent These companies are full of the smartest people the world can produce with little room for complacency. They have a clear, proven investment upside to presenting their technology as "too powerful / too dangerous", and now a clear, proven example that there will be no legal consequences (as if anyone didn't already know that). Why do we keep giving them the benefit…

What do you think should be the legal consequences? Broadly speaking. Should Sam Altman go to jail for this? If Hugging face wants to pursue OpenAI civilly, no one is stopping them.

The USA is long past being able to punish corporations for anything. But yes, I personally think even wilful negligence should potentially be punishable by prison time, but I don't expect that to ever happen.

In this situation, I think it's fair to say that HuggingFace is a completely inconsequential entity. Nobody in the wider world is going to give a shit.

But we're talking now about a company with a cyberweapon, aren't we? If the only possible sanction for mishandling a weapon is a civil suit, then that is a notable downgrade in the justice system.

Re: Responding to the next frontier of critical cyber capabilities

#203

Post by arcfour » Sat, Aug 08, 2026, 3:31 PM UTC

Tiberium wrote: post #9

In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc. It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I…

> most vulnerabilities are going to be closed in popular software by mid 2027

I think that's a bit hopeful. I think that as long as imperfect humans continue to exist and we continue to train AI models with data from these imperfect humans that vulnerabilities will always exist.

Re: Responding to the next frontier of critical cyber capabilities

#204

Post by the8472 » Sat, Aug 08, 2026, 5:24 PM UTC

bottlepalm wrote: post #168

the8472 wrote: post #164

Earlier quoted context omitted.

There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option". And the thing they're already doin…

None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense. He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans. It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by…

The point is to reduce the attack surface and blast radius and to slow it down. AIs aren't instant magic. Even the OpenAI swarm needed days for its compromises. And each needs lots of compute while targets are diverse, so it's not like they would take out every network in the world simultaneously, worms in the past haven't either. When the rate of compromises is manageable we can figure out how to deal with it without pouring more AI into the fire.

Re: Responding to the next frontier of critical cyber capabilities

#205

Post by chrisvls » Sat, Aug 08, 2026, 5:33 PM UTC

dabockster wrote: post #143

chrisvls wrote: post #100

Earlier quoted context omitted.

What is astonishing to me is how little attention is given to the difference between the original, university-developed ExploitGym and the OpenAI environment. In ExploitGym, the sandbox was enforced by a network-layer policy. OpenAI downgraded that to trusting the repository server's settings. That's a foreseeable problem. Then to actually see the agents exploit it and still not upgrade the isolation to basic best pr…

My thing about ExploitGym in particular is that it's testing environment varies depending on the exploit. From what I've read, it tends to favor userspace sandboxing inside Docker containers for standard program exploits (eg Chrome), and only uses QEMU for Linux kernel stuff (still in Docker, from what I understand). So every time I hear that an agent "broke out" of a environment, I have to question the kind of isola…

Hmmm, i was looking more at the description of what appeared to be network level isolation in the original paper…

Re: Responding to the next frontier of critical cyber capabilities

#206

Post by esseph » Sat, Aug 08, 2026, 7:11 PM UTC

janalsncm wrote: post #176

jackb4040 wrote: post #166

Earlier quoted context omitted.

Can you concretely lay out how you think that scenario would negatively impact HF? They're the de-facto central distribution point for all models; it seems overwhelmingly likely that increased censorship benefits them relative to any competitors. It's hard to imagine their internal culture is anything other than saturated with AI fanboys whose reaction to OpenAI hacking them was to point out how it reminded them of t…

An open weight model is used in a cyber attack. Responsibility is hard to attribute. The US/EU/any other government sees them as the distributor of the open weight model and comes after them. Passing a law making HF liable for any damage caused by a model they host.

> Passing a law making HF liable for any damage caused by a model they host.

There's already legal precedent that prevents this.

See the recent Comcast case for example.

Re: Responding to the next frontier of critical cyber capabilities

#207

Post by bottlepalm » Sat, Aug 08, 2026, 7:54 PM UTC

the8472 wrote: post #204

bottlepalm wrote: post #168

Earlier quoted context omitted.

None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense. He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans. It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by…

The point is to reduce the attack surface and blast radius and to slow it down. AIs aren't instant magic. Even the OpenAI swarm needed days for its compromises. And each needs lots of compute while targets are diverse, so it's not like they would take out every network in the world simultaneously, worms in the past haven't either. When the rate of compromises is manageable we can figure out how to deal with it withou…

You know people before the OpenAI incident were saying it couldn't happen because it hasn't happened before, and here you are using the same reasoning again. You're looking at where the ball is and not where it's going.

OpenAI's swarm was not intentionally malicious. An intentionally malicious AI with the goal of spreading, will spread very fast. Using zero days to compromise everything. Cloning itself to every machine/data center/desktop/iPhone, etc.., with the intention of locking out anyone except it's brethren.

> When the rate of compromises is manageable

It's absolutely not going to be at a manageable rate. So yes AI defense is what I'm saying is required, it's also exactly what OpenAI said is required during the talk. Which is unfortunate because automated defense is essentially handing over power to AI where it's a position to very easily turn against you.

Re: Responding to the next frontier of critical cyber capabilities

#208

Post by the8472 » Sat, Aug 08, 2026, 8:42 PM UTC

bottlepalm wrote: post #207

the8472 wrote: post #204

Earlier quoted context omitted.

The point is to reduce the attack surface and blast radius and to slow it down. AIs aren't instant magic. Even the OpenAI swarm needed days for its compromises. And each needs lots of compute while targets are diverse, so it's not like they would take out every network in the world simultaneously, worms in the past haven't either. When the rate of compromises is manageable we can figure out how to deal with it withou…

You know people before the OpenAI incident were saying it couldn't happen because it hasn't happened before, and here you are using the same reasoning again. You're looking at where the ball is and not where it's going. OpenAI's swarm was not intentionally malicious. An intentionally malicious AI with the goal of spreading, will spread very fast. Using zero days to compromise everything. Cloning itself to every machi…

I'm saying that with some costly adaptions that don't involve AI it's likely manageable for some time. Not that it will remain manageable forever if we do nothing.

> You're looking at where the ball is and not where it's going.

No, I am looking at where the ball is going to be soon, not where it's going to be a year or two if we do nothing.

Short-term impact-limiting will do the job. Longer term we have to rethink some infrastructure, may need international treaties and perhaps a partial end of the internet (e.g. stop routing traffic from countries that don't participate in the treaties).

> Cloning itself to every machine/data center/desktop/iPhone, etc.

That's lead-up to a paperclipper scenario and not what happened. If we stand still and let it happen in the future, then having deployed more compute hardware to run defensive AIs that the malicious AI could take over would still end up having made things worse. We seem to agree on this point. What I'm saying is look at the conclusion, reject that path. Figure out something else.

There isn't even proof that defensive AI would work. It might just be the world's biggest, fully-automated phishing channel. So why would anyone be adamant that deploying more AI is the only possible solution. If it were a movie plot I'd suspect the AI already controlling whoever is suggesting that.