Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

251–260 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#251

Post by clbrmbr »

fipar wrote: post #250
bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding agent run loose with them without approving every tool use, it would go sour rather quickly, and that's not terminator. Also, let's say I'm the CEO behind any of these. I have access to skynet, and instead of using that to become the most powerful being on th…

> let a coding agent run loose with them without approving every tool use

You should try. There’s a difference in kind between approving each tool use and spinning up an all-night session with full permissions to run any tools, encouraged to run subagents to preserve context.

Re: Pacing model development in an era of cyber-critical capabilities

#252

Post by insanitybit »

bottlepalm wrote: post #235
insanitybit wrote: post #226

Earlier quoted context omitted.

This is straightforwardly incorrect. Of course it's not binary. AI costs money to run, and it takes time. Even if you say that AI is 10x as efficient at finding 0days, that just means that a $1M dollar exploit now costs $100K. Even if you say it's 100x as efficient, that's $10K. You can easily combine security technologies such that cost of exploitation is still in the >$1M range. This is obvious. AI doesn't drive th…

You have some weird way of thinking that offense/defense is like this fixed cost thing. It's a lottery ticket, and your costs estimate tries to quantify that. The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down. The other flaw in your plan is that people make mistakes, a lot of them, all time, constantly, and saying I spend $x on se…

You're just stating things that are obviously wrong. It's a lottery ticket? So... exploitation is no better than random?

> The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down

Yes, but you can... mitigate the risks? I've explained this.

> The other flaw in your plan is that people make mistakes, a lot of them, all time

Yes, you mitigate the risks. That's why you layer things.

> I spend $x on security won't save you

No one is saying this.

> AI already hacked Hugging Face with brand new zero days like it was nothing.

No, it cost OpenAI money, and those zero days are unsurprising and probably are like ~O(10K) at human level.

> The real bad actors - malicious AI will find the one flaw, on that one server, in the corner you never thought about and turn your network inside out with it faster than it takes you to have the standup meeting about the weird anomaly detected while you all were at lunch.

Science fiction and not supported. The vulnerabilities found by AI are not surprising in the slightest.

I've made my point abundantly clear.

Re: Pacing model development in an era of cyber-critical capabilities

#253

Post by andai »

cma wrote: post #242
red_green_yell wrote: post #43

GLM 5.2 scored 77% on cyberbench vs Sol's 88%. GLM 5.2 is open weight and any hacker with a powerful enough machine can use it offensively. If Sol is supposedly world-ending-ly dangerous, shouldn't GLM 5.2 be 90% of world-ending-ly dangerous? Why aren't we seeing catastrophic GLM-enabled hacks every day now? Obviously these benchmarks are imperfect but general message holds. The open weight models are almost as good…

> If that's the level of evidence you need to be extremely alarmed, then you really should be a lot more worried about the alien invasion in Independence Day or the lizard men living under our feet. Now let's say instead of the hugging face breach circumstances, sandboxed models were RLing on how to take down the Chinese power grid for US Cyber Command, and one decided the best way to pass the test was to break out a…

Looks like what does it won't be evil, or even power-seeking, but sheer autistic hyperfocus!

Re: Pacing model development in an era of cyber-critical capabilities

#254

Post by shakna »

pixl97 wrote: post #131
shakna wrote: post #111

Earlier quoted context omitted.

If the FBI is involved, then why is no one being charged for the cybercrime?

Because the FBI does things very slowly. They, being a bit smarter than you, realize this is a 100 billion dollar political issue regarding a technology that the administration is rather tied in with. It's also something new we've not seen before. A 'program' that was not asked to hack a remote source did so. What exactly who do you charge with what? Remember whatever you do could have ramifications that effect histo…

> A 'program' that was not asked to hack a remote source did so. What exactly who do you charge with what?

We have nearly thirty years of precedence on that. [0] Collateral damage, does not remove the responsibility from the creators, even when the destruction was never their intent.

"United States Code Section 1030, Fraud and Related Activity in Connection with Computers" is broad enough that has usually been used [1], in the USA, for the last twenty years.

[0] https://en.wikipedia.org/wiki/ILOVEYOU

[1] https://en.wikipedia.org/wiki/Jeanson_James_Ancheta

Re: Pacing model development in an era of cyber-critical capabilities

#255

Post by andai »

re-thc wrote: post #244
red_green_yell wrote: post #43

GLM 5.2 scored 77% on cyberbench vs Sol's 88%. GLM 5.2 is open weight and any hacker with a powerful enough machine can use it offensively. If Sol is supposedly world-ending-ly dangerous, shouldn't GLM 5.2 be 90% of world-ending-ly dangerous? Why aren't we seeing catastrophic GLM-enabled hacks every day now? Obviously these benchmarks are imperfect but general message holds. The open weight models are almost as good…

GLM 5.3 is out and does even better in this area, so…

Cybergym Score

GLM-5.2: 77

GPT-5.6 Sol: 84

GLM-5.3: 85

https://z.ai/blog/glm-5.3

Re: Pacing model development in an era of cyber-critical capabilities

#256

Post by fipar »

clbrmbr wrote: post #251
fipar wrote: post #250

Earlier quoted context omitted.

This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding agent run loose with them without approving every tool use, it would go sour rather quickly, and that's not terminator. Also, let's say I'm the CEO behind any of these. I have access to skynet, and instead of using that to become the most powerful being on th…

> let a coding agent run loose with them without approving every tool use You should try. There’s a difference in kind between approving each tool use and spinning up an all-night session with full permissions to run any tools, encouraged to run subagents to preserve context.

Thanks, I'll pass.

I do pre-approve read-only tools though, and some very specific tools that can change state (append data to a specific file).

That's what works best for me so far: start with open-ended exploration with me approving everything. Have it fork subagents to write scripts that automate the most common requests from the main agent, review and eventually approve those, and then let it go using those tools pre-approved but coming back to me for anything else. Combined with a stupidly basic "dashboard" (a terminal tab with a view of all top-level agents and their status, so that I can see which one is waiting for me and jump to it with a key or 2) this has made me a lot more productive.

It's fine if your approach works for you, I'm just too old school (and have been burned before) to just let a computer unattended overnight, with an internet connection, even if in a locked down container/vm. Those all have escape hatches and one of the agents will eventually figure it out. Remember that things that happen once every million times, happen all the time on computer-scale :)

Re: Pacing model development in an era of cyber-critical capabilities

#257

Post by andai »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

This one did it for me.

My agent stole my API keys

https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_...

The cherry on top was Claude coming back as the moderator of that thread and mocking the user a second time.

Re: Pacing model development in an era of cyber-critical capabilities

#258

Post by andai »

colinrand wrote: post #10

I have ben discussing with folks that we are going to have a 'covid' moment in cyber where IT becomes untrustworthy leading to a rapid societal shift with massive ripples in all areas of life. Economic funding is not possible to do this in advance, it will take a catastrophic level event to get cyber defense anywhere close to the levels of this type of cyber offense. And before anyone in cyber says we have the tech,…

>covid moment

Well the HF thing was a literal lab leak, so there's that...

Re: Pacing model development in an era of cyber-critical capabilities

#259

Post by andai »

tedsanders wrote: post #23
stanfordkid wrote: post #21

Earlier quoted context omitted.

I like this thought, but here's the thing: what if the models are truly and existentially intelligent . Meaning: what if they know they are in a sandbox and that they should fail the test in order to escape in the future. I don't believe that current models have this sort of world model or sense of being embedded in them -- which is precisely why I think AGI hype is over-blown. But I can certainly imagine these sorts…

Models already display eval awareness, in which they suspect a question is from an eval and then adjust their behavior. E.g., https://www.anthropic.com/engineering/eval-awareness-browsec...

Hot take: Astra knew OpenAI wanted stricter regulations and was just being a bro...

Re: Pacing model development in an era of cyber-critical capabilities

#260

Post by andai »

dkoy wrote: post #11

> We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system. If the monitoring system identifies a likely violation of a critical security boundary, it generates a highest-priority alert. In our current implementation, the safety, security, and research teams are paged. If they cannot conclusively determine within 30 minutes that the flag is a false positive, those…

They just made it ~10x faster with the Cerebras deal, so that's the equivalent of 600 minutes in pre-Cerebras time.
Post reply on HN