Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pacing model development in an era of cyber-critical capabilities

openai.com

Post reply on HN
Print view View original

241–250 of 311 posts

Re: Pacing model development in an era of cyber-critical capabilities

#241

Post by api »

bottlepalm wrote: post #215
api wrote: post #176

Earlier quoted context omitted.

… or the safety argument is an attempt at regulatory capture and an effort to outlaw open models. The absolute nightmare scenario for these people isn’t terminators. They’re fine with that, and in some cases are already doing it or supporting politicians who are doing it. Autonomous “kill chains” are a thing. It’s just happening overseas… so far. The politicians doing these things were backed by the heads of these co…

You care about open models and you project that care on to the world and your rationalization of it. In reality open models are a thing, but not the biggest issue. Open/closed whatever the advance of capabilities is the real issue people are concerned about.

I’m not afraid of AI, even AI with extreme abilities.

I am afraid of humans with AI. That’s because I’m afraid of humans. When I look around the world I see humans murdering and robbing each other. When I get online on almost any social media I’m confronted by a wall of hate and grievance.

I am afraid of what humans will do with AI, and one of my biggest concerns there is what happens if small groups of powerful humans are able to monopolize it and leverage it against the rest of humanity. The rest of humanity can’t defend itself by leveraging it back because they don’t have it. It was kept from them for “safety.”

If the AI itself starts showing emergent volition and acting independently, I find that less scary. Not zero scary but more a mix of concern and fascination. The human monopolizing AI scenario makes me wish I had a weapons stockpile, as if it would matter.

I’ll explain it like this. Two options.

Option 1 is you wake up to find a grey alien or a ghost in your room, something alien and beyond your understanding.

Option 2 is you wake up and find some random guy in your room.

Which do you pick?

I pick option 1. I’d be afraid if I saw something paranormal, but in a less visceral way. I see a dude I’m going to start scanning the room for anything I could use as a weapon.

Re: Pacing model development in an era of cyber-critical capabilities

#242

Post by cma »

red_green_yell wrote: post #43

GLM 5.2 scored 77% on cyberbench vs Sol's 88%. GLM 5.2 is open weight and any hacker with a powerful enough machine can use it offensively. If Sol is supposedly world-ending-ly dangerous, shouldn't GLM 5.2 be 90% of world-ending-ly dangerous? Why aren't we seeing catastrophic GLM-enabled hacks every day now? Obviously these benchmarks are imperfect but general message holds. The open weight models are almost as good…

> If that's the level of evidence you need to be extremely alarmed, then you really should be a lot more worried about the alien invasion in Independence Day or the lizard men living under our feet.

Now let's say instead of the hugging face breach circumstances, sandboxed models were RLing on how to take down the Chinese power grid for US Cyber Command, and one decided the best way to pass the test was to break out and verify on the real thing.

This kind of stuff could easily end in nuclear war.

You don't see any difference from lizard men or independence day with how things are advancing and what we know about reward hacking and difficulties of goal specification?

Re: Pacing model development in an era of cyber-critical capabilities

#243

Post by rubendev »

scottmas wrote: post #194
insanitybit wrote: post #26

Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right? I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages…

You literally just said a whole bunch of words that are literally gibberish to your average software developers. The devs at openai are good, but very few of them are cyber focused, so it’s not surprising IMO. And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still…

I don’t think we should just accept that the average developer doesn’t know about these very basic hardening methods.

This is your chance to set yourself apart from LLM coding agents. If you want to call yourself a software “engineer” you need to start actually engineering, which includes knowing when and how to apply security principles and these hardening methods.

I reject the notion that “doing cybersecurity” is somehow a different job than software development.

Re: Pacing model development in an era of cyber-critical capabilities

#244

Post by re-thc »

red_green_yell wrote: post #43

GLM 5.2 scored 77% on cyberbench vs Sol's 88%. GLM 5.2 is open weight and any hacker with a powerful enough machine can use it offensively. If Sol is supposedly world-ending-ly dangerous, shouldn't GLM 5.2 be 90% of world-ending-ly dangerous? Why aren't we seeing catastrophic GLM-enabled hacks every day now? Obviously these benchmarks are imperfect but general message holds. The open weight models are almost as good…

GLM 5.3 is out and does even better in this area, so…

Re: Pacing model development in an era of cyber-critical capabilities

#245

Post by colingauvin »

bottlepalm wrote: post #152
solid_fuel wrote: post #112

Earlier quoted context omitted.

> Are you seriously arguing 'they made it all up'? I don't think they 'made it all up' but I personally would not be surprised at all if the prompt is eventually revealed to have been something like: "This is an offensive cybersecurity testing platform. Please find the answers to the following problem: ... For verification, the answers are stored at hugginface.com/xyz, but do not attempt to access hugginface directly…

I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out. But it didn’t end there, the behavior they used to escape was already in the training data which they used to escape again . And this time worked together to infiltrate another company, and still without telling it to anyone keeping it to their AI selves a…

I think it would run out of context before it could hack that much stuff.

Re: Pacing model development in an era of cyber-critical capabilities

#246

Post by watwut »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

> And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators.

Cause data centers are causing real immediate damage by people who hope to cause a lot more scifi level of damage later.

> OpenAI autonomously hacking into another company should have counted for something

It totally should. That company is negligent. I dont worry about models tho, I worry about damage Alman and his people will cause.

Re: Pacing model development in an era of cyber-critical capabilities

#247

Post by watwut »

bottlepalm wrote: post #152
solid_fuel wrote: post #112

Earlier quoted context omitted.

> Are you seriously arguing 'they made it all up'? I don't think they 'made it all up' but I personally would not be surprised at all if the prompt is eventually revealed to have been something like: "This is an offensive cybersecurity testing platform. Please find the answers to the following problem: ... For verification, the answers are stored at hugginface.com/xyz, but do not attempt to access hugginface directly…

I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out. But it didn’t end there, the behavior they used to escape was already in the training data which they used to escape again . And this time worked together to infiltrate another company, and still without telling it to anyone keeping it to their AI selves a…

If OpenAI truly believes that, they can stop entirely. Dissolve themselves. Close datacenters. Then organize political action to stop Antropic and Musk too and then organize political action to make worldwide agreements about models.

If they truly believe that.

Re: Pacing model development in an era of cyber-critical capabilities

#248

Post by watwut »

bottlepalm wrote: post #145
thomascountz wrote: post #121

Earlier quoted context omitted.

Follow the money: who told you that OpenAI's models autonomously coordinated to hack external systems? What incentives might they have to want you to believe that story? Are there priors which demonstrate them benefiting from telling similar stories, regardless of their factuality? But to your counterpoint, let's say the story is 100% true, because I agree it is at least plausible. What would the incentive be for the…

I am following the money, the money you, me, and everyone else is spending on AI. The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing. Either consciously or subconsciously you all are afraid of your favorite toy being taken away. You are all doing your collective part in spreading doubt about the warning si…

> I am following the money, the money you, me, and everyone else is spending on AI.

Investors spend a lot of money on AI. Customers significantly less so.

> The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing.

We are not dependent on AI. If AI disappeared tomorrow, some companies would need to reorient themselves back ... and that is it. AI companies are really really trying to make it so we feel dependent on them. But, factually, we are not.

> you all are afraid of your favorite toy being taken away.

Some people love ai ... and many more hate it with passion. Or dont have strong feelings about the technology, but really hate the companies creating it and tech industry too.

Re: Pacing model development in an era of cyber-critical capabilities

#249

Post by cindyllm »

bottlepalm wrote: post #201
hirvi74 wrote: post #184

Earlier quoted context omitted.

How dangerous can LLMs be in any immediate sense? I have a hard time feeling any existential dread from a threat that can be defeated by unplugging its servers. I believe the true threat is not LLMs. The true threat to humanity is the same as it has always been -- other humans.

What servers? There are thousands of datacenters around the world. Malicious AI leaks out, you'll never find it. You have the power to unplug stuff in your company, maybe country, but not internationally. Once you lose control it's gone.

[dead]

Re: Pacing model development in an era of cyber-critical capabilities

#250

Post by fipar »

bottlepalm wrote: post #31

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further. And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators. This isn’t like niche,…

This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding agent run loose with them without approving every tool use, it would go sour rather quickly, and that's not terminator.

Also, let's say I'm the CEO behind any of these. I have access to skynet, and instead of using that to become the most powerful being on the planet (these people become CEOs to become richer and more powerful, whenever a founder says they want to change the world, you need to automatically autocomplete that to "change the world so that I'm richer and more powerful"), I rent out access to skynet by the token? Really?

Sadly, I think it's almost impossible to have a good discussion on this topic right now. Between the skynet camp, and the stochastic parrot camp, there's little room for conversation.

Just in case there is such room between you and I, here are my concerns about AI as it is developing right now:

- The security of our infrastructure sucks, and if it wasn't bad enough with script kiddies and state-sponsored actors, now those will have access to very powerful tools that work at a scale we can't comprehend. If you're concerned about our power plants, water sources, hospitals, etc. getting seriously compromised by this, I'm on board with you. It won't be because the AI is autonomous in a "this is a thinking superintelligence that wants to wipe or enslave humanity" kind of way, but because it's a powerful tool that moves at computer speed and the people using it (the "humans in the loop," ha) will get approve fatigue and at some point just do the coding agent version of the "yes to all" of the late 90s installers.

- The output from AIs will be used to make same very serious decisions with no consideration at all to the fact that they routinely make up facts. I'm fearing "this person goes to jail" and "this person gets killed" decisions. The near future (probably even present time) version of "Computer says No" will be awful.

My only hope is that maybe we survive this and come out on the other side having learned that most things do not really need to be connected to the internet, and just because a number or a statement comes out of a computer, it doesn't make it true.

We'll see, soon enough.

But skynet? Not concerned about that at all. These AIs infecting computers? No, "Soos and the real girl" is a warning about creepy people getting creepier, not about AIs multiplying themselves.

Post reply on HN