Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Responding to the next frontier of critical cyber capabilities

openai.com

Post reply on HN
Print view View original

101–110 of 211 posts

Re: Responding to the next frontier of critical cyber capabilities

#101

Post by boie0025 »

Legend2440 wrote: post #43
reducesuffering wrote: post #30

Every fifth comment about our insane trajectory of AGI is about "marketing." These incidents and cybersecurity capabilities are now involving government hearings and the CIA. Denial is truly an incredible thing in the face of a very scary immediate future.

A whole ton of people desperately want to believe that LLMs are a lie that will be revealed as a scam... any day now.

I agree with this assessment. My consulting business partner and I are in the process of moving a bunch of applications from an old deployment managed with Ansible on EC2 over to an ECS deployment. We're doing this app by app, and it's taking a while. The last two apps we both moved we were able to point the model at the appropriate repository and aws account and it was able to execute the entire move on both of these _in one shot_ with various CLI tools. That's not a parlor trick.

Re: Responding to the next frontier of critical cyber capabilities

#102

Post by jackb4040 »

reducesuffering wrote: post #30

Every fifth comment about our insane trajectory of AGI is about "marketing." These incidents and cybersecurity capabilities are now involving government hearings and the CIA. Denial is truly an incredible thing in the face of a very scary immediate future.

We live in the era of Project Stargate. We live in the era of military insiders getting caught rigging Polymarket. We live in the era where AI companies are the only reason we're not in a recession. We live in the era of DOGE and Kash Patel.

I don't think there's too many people who distrust AI companies but trust the current government or the CIA as impartial authorities. I'm not saying you don't have an argument, but appeals to that specific authority will not be effective except among people who already agree with you.

Re: Responding to the next frontier of critical cyber capabilities

#103

Post by chrisvls »

hluska wrote: post #45
magicalist wrote: post #27

Earlier quoted context omitted.

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about. If you consider that incompetence, it’s possible that you’re not a very nice person.

The original ExploitGym used network-layer isolation. OpenAI downgraded that to trusting the repository server's settings. Giving the agent that kind of attack surface seems like a pretty foreseeable problem. I don't think it was intentional, but it's not a stretch to say that they aren't good enough at this job.

Re: Responding to the next frontier of critical cyber capabilities

#104

Post by NitpickLawyer »

ofjcihen wrote: post #69
jrflo wrote: post #53

Earlier quoted context omitted.

Not a security guy but my understanding is: you only need to find one flaw to exploit a system, to make a system totally secure you need to find them all. It's inherently easier to use these tools offensively rather than defensively.

I’m a cybersecurity guy. >” you only need to find one flaw to exploit a system” I see this everywhere, especially in these threads and it’s not even remotely true for modern architecture. Between principles like zero-trust, defense in depth, etc. we’ve been away from the one flaw situation for a long time. Now does crap software exist that doesn’t follow these principles? Absolutely. But those were a problem before A…

They do address some of these things in the final slides / "lessons learned" section of the defcon talk. Good security practices will continue to be good, but... and there are a lot of buts here.

I disagree with your take that "it's not even remotely true" and "we've been away from...". We really really haven't. This is as true as it has always been. Any system is as secure as the weakest link. That link can be anything from a human, to a leaked token, to a badly configured server, to bad code running somewhere. The amount of leaks / ransomware attacks / etc in the past 5-10 years serve as ample evidence.

And now, right now, there are "red team" capabilities that can literally bang tokens against the wall until they find that weakest link, and then can move laterally with inhuman speed. That's the reality, now. The "blue team" capabilities are lacking, because the bottleneck is with humans. From alert fatigue, to not enough trained people, to having to vet every new RCE, to having to test, deploy and validate any mitigations, the scales are currently favouring the automated side.

Re: Responding to the next frontier of critical cyber capabilities

#105

Post by jackb4040 »

mephux wrote: post #15

We all know this is propaganda to get a gov bailout or to slow down competition with regulations right? If this was an issue companies that did red team engagements would have been regulated long ago. There is no regulations on companies that actively exploit products, companies and services for money. They could all be bad actors.. yet, no regulations.. its all nonsense. If it's important.. don't connect to the inte…

Don't let the downvotes get you down. We are right, and as these companies get more desperate the shrinking minority that still wants them to succeed will only get louder.

Re: Responding to the next frontier of critical cyber capabilities

#106

Post by perching_aix »

meatmanek wrote: post #91
dboreham wrote: post #63

By "cyber" they mean "cybersecurity".

Yeah, this irks me. It's bad enough that the LLMs themselves are changing our language by tainting certain words/phrases/patterns as LLM-coded; now the companies themselves have decided that they just get to synecdoche the word/prefix "cyber".

This is not actually a neologism, it predates the LLM era.

Re: Responding to the next frontier of critical cyber capabilities

#107

Post by ameliaquining »

jackb4040 wrote: post #98
AlotOfReading wrote: post #58

Earlier quoted context omitted.

Isolation is a pretty standard requirement for lots of networks, especially gov. AWS provides isolation domains, and used to have a unidirectional cross-domain service called AWS diode. Or you can just configure your gateways properly / buy data diodes. Lots of possible solutions.

They can't have it both ways. You don't get to tell the media your product is more dangerous than nuclear weapons for precisely this reason , and then do less to secure it than an off-the-shelf AWS product that predates LLMs.

Nobody is claiming that any currently-existing LLM is more dangerous than nuclear weapons.

Re: Responding to the next frontier of critical cyber capabilities

#108

Post by orthogonal_cube »

AlotOfReading wrote: post #58
ducktective wrote: post #7

Earlier quoted context omitted.

> including isolated testing environments Given the attack vector having possible super-human capability, I'm not sure such an environment exists. "Isolated" according to who? Maybe seL4 could be a viable option here...

Isolation is a pretty standard requirement for lots of networks, especially gov. AWS provides isolation domains, and used to have a unidirectional cross-domain service called AWS diode. Or you can just configure your gateways properly / buy data diodes. Lots of possible solutions.

There’s still levels to it.

“Isolation” can mean the network hardware has no direct connections to an extranet. Data is transferred manually by physical media (USB, DVD, etc.) with logging and dedicated transfer stations.

“Isolation” can mean a VLAN on equipment which has also has extranet access, creating a logical isolation rather than physical (to reduce cost). Data can be transferred manually or through diodes.

And then there’s “isolation” which is a joke: machines technically able to access the internet but require proxy configuration (which isn’t set but can be easily derived).

Re: Responding to the next frontier of critical cyber capabilities

#109

Post by scarmig »

ofjcihen wrote: post #32
magicalist wrote: post #27

Earlier quoted context omitted.

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

Right? Like I feel like I’m taking crazy pills. OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”

"The fire fighting department really just hired an arsonist so they could market their fire fighting abilities!"

Re: Responding to the next frontier of critical cyber capabilities

#110

Post by faidit »

magicalist wrote: post #27
NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

Or they wanted it to happen again. Worst case outcome, the US Congress spontaneously grow backbones, spite their sponsors and unite in a bipartisan effort for effective regulation because a few people got hacked? Likelihood low - the hacked companies might even oppose any meaningful AI regulation because it hurts their inflated profit forecasts. More probable outcomes: Useful real-world testing, free headlines about AI breakthroughs, scare USG into providing more free money ("look how scary it is - what if China develops this faster than us?")
Post reply on HN