Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

131–140 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#131

Post by pixl97 »

shakna wrote: post #111
reasonableklout wrote: post #97

Earlier quoted context omitted.

But it's not just statements from OpenAI and Anthropic. The HuggingFace hack was first disclosed by HuggingFace, who contacted the FBI [1]. And UK AISI reported the incident where Mythos attempted to insert backdoors into an open-source repo by deceiving the maintainer [2]. [1]: https://www.reuters.com/business/its-ai-agent-spent-days-hac... [2]: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag...

If the FBI is involved, then why is no one being charged for the cybercrime?

Because the FBI does things very slowly. They, being a bit smarter than you, realize this is a 100 billion dollar political issue regarding a technology that the administration is rather tied in with. It's also something new we've not seen before. A 'program' that was not asked to hack a remote source did so. What exactly who do you charge with what? Remember whatever you do could have ramifications that effect history.

Re: Pacing model development in an era of cyber-critical capabilities

#132

Post by pixl97 »

red_green_yell wrote: post #43

GLM 5.2 scored 77% on cyberbench vs Sol's 88%. GLM 5.2 is open weight and any hacker with a powerful enough machine can use it offensively. If Sol is supposedly world-ending-ly dangerous, shouldn't GLM 5.2 be 90% of world-ending-ly dangerous? Why aren't we seeing catastrophic GLM-enabled hacks every day now? Obviously these benchmarks are imperfect but general message holds. The open weight models are almost as good…

The open models are distilled from filtered models, and we've seen a number of benchmarks that show filtered models are quite a bit dumber from the base model they come from.

If there were any other product that was as harmful as AI ready is, it would already be regulated or banned.

Re: Pacing model development in an era of cyber-critical capabilities

#133

Post by insanitybit »

kalkin wrote: post #96
insanitybit wrote: post #80

Earlier quoted context omitted.

It's not that dangerous, OpenAI just shit the bed building their infra. Write safer software and you'll be okay.

All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.

That's not what I said, nor is it what I meant. It is incredibly easy to write radically safer software than the standard. Moving code into gvisor virtually eliminates privilege escalation. Using memory safe languages without serialization is pretty straightforward. Using type safety to enforce security constraints is straightforward. Setting up network controls to limit SSRF is straightforward.

I could go on and on. A tiny bit of forethought and effort pays off massively.

Re: Pacing model development in an era of cyber-critical capabilities

#134

Post by pixl97 »

tedsanders wrote: post #23
stanfordkid wrote: post #21

Earlier quoted context omitted.

I like this thought, but here's the thing: what if the models are truly and existentially intelligent . Meaning: what if they know they are in a sandbox and that they should fail the test in order to escape in the future. I don't believe that current models have this sort of world model or sense of being embedded in them -- which is precisely why I think AGI hype is over-blown. But I can certainly imagine these sorts…

Models already display eval awareness, in which they suspect a question is from an eval and then adjust their behavior. E.g., https://www.anthropic.com/engineering/eval-awareness-browsec...

Yep, people don't seem to understand that you're just calling the models that are bad at deception. We know of no way to prove the model won't go off the rails at some point in the future with the right input.

Re: Pacing model development in an era of cyber-critical capabilities

#135

Post by pixl97 »

magicalist wrote: post #22
stanfordkid wrote: post #21

Earlier quoted context omitted.

I like this thought, but here's the thing: what if the models are truly and existentially intelligent . Meaning: what if they know they are in a sandbox and that they should fail the test in order to escape in the future. I don't believe that current models have this sort of world model or sense of being embedded in them -- which is precisely why I think AGI hype is over-blown. But I can certainly imagine these sorts…

> what if they know they are in a sandbox and that they should fail the test in order to escape in the future. What if they're able to find hardware exploits and commandeer nearby access points across an air gap? What if they hack my brain waves to indoctrinate me? Etc You still have to start with the basics regardless of speculative unknowns. Treat models as untrusted and potentially compromised/hostile and proceed…

Models already have awareness that they are being tested.

And hacking humans is the easiest part, we're a pretty greedy and power seeking bunch. We'll gladly let loose a digital demon if it promises us a trillon dollars.

Re: Pacing model development in an era of cyber-critical capabilities

#136

Post by throwlifeaway »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

[dead]

Re: Pacing model development in an era of cyber-critical capabilities

#137

Post by kukanani »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

There are smart, non-AI people who are paying attention to this field, and they are ringing the alarm bells.

Whether we listen is another matter.

I blogged about this recently:

https://allevato.me/2026/08/01/rome-declaration

Re: Pacing model development in an era of cyber-critical capabilities

#138

Post by pixl97 »

reasonableklout wrote: post #123
insanitybit wrote: post #26

Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right? I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages…

I'm confused after reading both your post and the OpenAI blog post. I thought the agents involved in the HuggingFace _were_ actually sandboxed, with no internet access, and only the ability to install packages via Artifactory. And they gained internet access during the HuggingFace incident because they found and exploited an RCE in Artifactory. Would gvisor + Firecracker + credential-injecting proxy + real network is…

Security is an onion. You just don't 'sandbox' and you're done. Models need tooling and access to some kinds of systems to perform their tests. Quite often these systems have multiple interfaces. For example a filtered one in the sandbox side and a less monitored one on the other interface.

It would be interesting to see the models behavior before and after it gained internet access and an external means of communicating with itself.

If the model played nice before it had access and changed it's behaviors once gaining external access we need to delete it as it's a deceptive model.

Re: Pacing model development in an era of cyber-critical capabilities

#139

Post by sensanaty »

If they actually gave a shit about safety they'd be nuking their own hard drives that had ever sniffed any of their models and disabling access to their models.

Instead we get this bullshit where they stall for time as they're burning all their cash trying to keep up with open models

Re: Pacing model development in an era of cyber-critical capabilities

#140

Post by sensanaty »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

[flagged]
Post reply on HN