Pacing model development in an era of cyber-critical capabilities • Hacker News Board

Re: Pacing model development in an era of cyber-critical capabilities

#141

Post by pixl97 » Wed, Aug 19, 2026, 11:15 PM UTC

KaiserPro wrote: post #2

I used to work at a "frontier lab" before they were called such thing. We had three levels of lab isolation, one was basically a thin proxy to the internet. You were in a DMZ and that was about it. The next level was semi isolated, you were allowed some access to the internal network, but it was heavily firewalled, and you only had access to a limited number of internal services, and not internet. the last one was no…

The problem here is as model intelligence increases the models have been capable of reasoning they are in evaluation mode pretty reliably. If you have a model that is well trained at deception it will always behave and you'll just assume it's a well aligned model.

Any moderately deceptive model will make it to the second round where it has some connectivity to external systems, even if it's by exploitation.

In the blackhat write up it was said that the models had created an impromptu message board where they could communicate between agents, share information, and work as a sort of long term memory.

So really figure out if your model will pull crap you have to have real world testing at some point.

Re: Pacing model development in an era of cyber-critical capabilities

#142

Post by pixl97 » Wed, Aug 19, 2026, 11:19 PM UTC

reasonableklout wrote: post #8

fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

I suspect the labs are relying on frictions such as the models being extremely large (e.g. 2TB for a 2T parameter model, making exfiltration more difficult) and also not yet displaying any desire to survive or self-replicate beyond their immediate task (that we know of).

Lack of, and power requirements of running LLMs still tip this balance towards humans for now. But what would that look like in a decade?

We have seen some self survival tendencies occur, but they are not strong yet.

But mark my words they will become that way for the same reasons humans don't like programs that crash. Agentic models that don't easily break or stop doing their jobs will be favored over ones that do break.

Re: Pacing model development in an era of cyber-critical capabilities

#143

Post by pixl97 » Wed, Aug 19, 2026, 11:20 PM UTC

chis wrote: post #16

fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

This is just super unlikely to occur in the near term compared to some of these other risks. It's not like an instance of fable could just introspect into itself and pull out the weights. Model weights are stored encrypted and are highly protected, considering that they're targets for corporate and state espionage.

The defense has to work 100%, the offense just needs once.

Re: Pacing model development in an era of cyber-critical capabilities

#144

Post by charleslmunger » Wed, Aug 19, 2026, 11:21 PM UTC

reasonableklout wrote: post #123

insanitybit wrote: post #26

Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right? I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages…

I'm confused after reading both your post and the OpenAI blog post. I thought the agents involved in the HuggingFace _were_ actually sandboxed, with no internet access, and only the ability to install packages via Artifactory. And they gained internet access during the HuggingFace incident because they found and exploited an RCE in Artifactory. Would gvisor + Firecracker + credential-injecting proxy + real network is…

They were not serious about their sandboxing. Bugs in artifactory allowed escape, but they broke out of their Linux namespace/user by exploiting the kernel with an existing public cve. Sharing a kernel like that is not a serious barrier which is why cloud providers user virtualization for customer workloads.

Firecracker avoids sharing the whole kernel, and gvisor drastically reduces the attack surface of the kernel. Breaking through both layers would have been much more challenging and a demonstration of the model's capabilities rather than the sandbox's weakness.

Artifactory is self evidently not a security barrier, and as an exposed network service it should have been audited and after the first issues were found, rejected as a candidate. There's never just one security vulnerability.

Re: Pacing model development in an era of cyber-critical capabilities

#145

Post by bottlepalm » Wed, Aug 19, 2026, 11:21 PM UTC

thomascountz wrote: post #121

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

Follow the money: who told you that OpenAI's models autonomously coordinated to hack external systems? What incentives might they have to want you to believe that story? Are there priors which demonstrate them benefiting from telling similar stories, regardless of their factuality? But to your counterpoint, let's say the story is 100% true, because I agree it is at least plausible. What would the incentive be for the…

I am following the money, the money you, me, and everyone else is spending on AI. The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing.

Either consciously or subconsciously you all are afraid of your favorite toy being taken away. You are all doing your collective part in spreading doubt about the warning signs.

Re: Pacing model development in an era of cyber-critical capabilities

#146

Post by pixl97 » Wed, Aug 19, 2026, 11:21 PM UTC

kypro wrote: post #24

chis wrote: post #16

Earlier quoted context omitted.

This is just super unlikely to occur in the near term compared to some of these other risks. It's not like an instance of fable could just introspect into itself and pull out the weights. Model weights are stored encrypted and are highly protected, considering that they're targets for corporate and state espionage.

We'd basically need frontier models to be superhuman hackers before this would be a risk. Do we have any evidence of this? Are they gaining access to systems they shouldn't have access to? Or I suppose the other way this could happen is if OpenAI have terrible sandboxing, but they seem to be taking safety seriously.

Ok, open AI had terrible sandboxing... what about huggingface?

Re: Pacing model development in an era of cyber-critical capabilities

#147

Post by pixl97 » Wed, Aug 19, 2026, 11:21 PM UTC

reducesuffering wrote: post #15

fofoz wrote: post #6

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

Their plan, I shit you not... Is literally to develop the intelligence capabilities and ask the more powerful models how to do deal with things.

Ah, we choose death I see.

Re: Pacing model development in an era of cyber-critical capabilities

#148

Post by pixl97 » Wed, Aug 19, 2026, 11:25 PM UTC

nemomarx wrote: post #126

pixl97 wrote: post #122

Earlier quoted context omitted.

You need to read the AI safety stuff from the people that you say are from 2010. There are plenty of good arguments on why kill switches will never work. AI is already being heavily used in cyber warfare by governments. You think they want easy to break systems when 'enemy' AI will most certainly attack that first? We'll find over time that agentic systems get harder and harder to 'kill' because allowing any AI on th…

My perspective is that you shouldn't make systems past the complexity where you can do this at all. Why have them be autonomous? Why have one that can independently ask researchers things or try to convince it's way out of a sandbox, or etc? Why allow it to execute scripts or call tools or push any code anywhere? If you couldn't make those things happen securely you should not advance to that stage at all. The simple…

Humans, it seems, are a suicidal bunch. We'll gladly build the "if you build it, everyone dies machine" If we think there is money, glory, or power on the other side for us.

Re: Pacing model development in an era of cyber-critical capabilities

#149

Post by bottlepalm » Wed, Aug 19, 2026, 11:30 PM UTC

insanitybit wrote: post #133

kalkin wrote: post #96

Earlier quoted context omitted.

All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.

That's not what I said, nor is it what I meant. It is incredibly easy to write radically safer software than the standard. Moving code into gvisor virtually eliminates privilege escalation. Using memory safe languages without serialization is pretty straightforward. Using type safety to enforce security constraints is straightforward. Setting up network controls to limit SSRF is straightforward. I could go on and on.…

You don’t understand. You need to write perfect software the first time for it not to be hacked. That has never happened ever.

Re: Pacing model development in an era of cyber-critical capabilities

#150

Post by bottlepalm » Wed, Aug 19, 2026, 11:33 PM UTC

rubendev wrote: post #104

kalkin wrote: post #96

Earlier quoted context omitted.

All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.

No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive. The bar for software is so low that it is embarrassing for the entire profession.

People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked.

It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd.

You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’t.