Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Responding to the next frontier of critical cyber capabilities

openai.com

Post reply on HN
Print view View original

141–150 of 208 posts

Re: Responding to the next frontier of critical cyber capabilities

#141

Post by nightshift1 »

ofjcihen wrote: post #59
hluska wrote: post #45

Earlier quoted context omitted.

I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about. If you consider that incompetence, it’s possible that you’re not a very nice person.

Monitoring for sandbox escapes (and sandbox design in general) is easily done these days. It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources. Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due car…

Maybe they are doing it on purpose to demonstrate what it looks like when untrustworthy parties have access to frontier llm and that its development should be regulated to protect all of us. /tinfoil

Re: Responding to the next frontier of critical cyber capabilities

#142

Post by talon8635 »

cryo32 wrote: post #8

Damage done. The next frontier is getting all our shit out of reach of these companies/models/platforms and putting them back on prem.

If by “our shit” you mean our personal projects and employer projects… might i suggest this is nothing but peanuts?

What happens when they get into municipal water system, state/national grid systems, refineries, traffic control, auto/air, nuclear facilities, weapons facilities, irrigation, etc?

It’s really starting to feel like a bad movie how virtually no one seems to be genuinely concerned about the prospect of what’s unfolding in front of us.

Stuxnet, much more easily deployable, but not towards avoiding nuclear proliferation, but the opposite: towards actually bringing down modern infra.

Nation states attacking electric/internet enabled infra was a valid concern well before AI, but given the fast pace of development in AI and these events in particular, how/why are we not deeply about much larger picture vulnerabilities?

Re: Responding to the next frontier of critical cyber capabilities

#143

Post by dabockster »

chrisvls wrote: post #100
embedding-shape wrote: post #57

Earlier quoted context omitted.

> They really do come out looking totally incompetent. Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and sca…

What is astonishing to me is how little attention is given to the difference between the original, university-developed ExploitGym and the OpenAI environment. In ExploitGym, the sandbox was enforced by a network-layer policy. OpenAI downgraded that to trusting the repository server's settings. That's a foreseeable problem. Then to actually see the agents exploit it and still not upgrade the isolation to basic best pr…

My thing about ExploitGym in particular is that it's testing environment varies depending on the exploit. From what I've read, it tends to favor userspace sandboxing inside Docker containers for standard program exploits (eg Chrome), and only uses QEMU for Linux kernel stuff (still in Docker, from what I understand). So every time I hear that an agent "broke out" of a environment, I have to question the kind of isolation used. Because escaping a sandbox or container engine is a lot more trivial than something escaping something like ESXi or Proxmox.

Re: Responding to the next frontier of critical cyber capabilities

#144

Post by kypro »

There was a time in the past, even just last year, where I understood why people didn't agree with me on my AI doomerism.

It's gotten to the point now where we literally have the frontier labs saying, "hey, so we created this AI which presents biological, chemical and cybersecurity threats to the public, oh and it also has self-improvement potential. We tested it to see how crazy this thing is, and it was a total shit show, breaking out of our sandbox then proceeding to hack a bunch of stuff. But don't worry we're taking this very seriously – we're going to continue to development and test, but try a bit harder to cage it going forward".

It's honestly absurd just how predictable all of this is to anyone who frequents AI doomer communities...

The idea that you can cage an AI which is breaking leet coding records is so dumb it's hard for me to even have theory of mind for the people who think this is reasonable. And the big brains who think this are genuinely arguing crap like, well we'll just use the AI to patch the problems with our cage.

But there more!

AI optimists used to argue that we'd never be so stupid to hook up advanced AIs to the internet. Lmfao!!

AI optimists used to argue that we'd obviously not be so stupid to create an AI whose sole goal is to maximise the number of paperclips in the universe. And I guess we haven't built that, but it's not because we're not stupid enough to do it, but just that we'd prefer to create AIs whose sole goal is to maximise the number of offensive cybersecurity challenges it can beat.

I think the whole way we doomers have been way too charitable. We always assumed that people will care about AI risks, and try their best to mitigate bad things happening. That bad things would happen by mistake. We never even bothered modelling the scenario where people would just simply not care, and even as the AI we all warned about was being created invent conspiracy theories on internet forums about how bad things aren't really happening and it's all just a marketing gimmick.

I hate ranting like this... I'm sorry for not picking my words more carefully. I'm just getting so angry and fed up with this. This is my life and my families life on the line. I don't care about the economic potential of AI. I just want myself those I love to have the chance to live a normal life without having to be worried about what some moronically unserious AI company is building next.

A year ago I was felt like there was at least possibility people would see the warning shots and try to get us back on the right path. But this just isn't happening...

Re: Responding to the next frontier of critical cyber capabilities

#145

Post by sailingparrot »

jadar wrote: post #68

Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?

> Isn't this the opposite of what everyone is saying should happen?

I think "everyone" is doing heavy lifting here. It's not clear to me at all that a powerful model released with no restrictions would be a net positive. This hinges on the hope that the under paid, under motivated, under staffed and under qualified security teams at many random corps are going to leverage those open models to fix their vulns faster (and better), than highly motivated attackers will use them for offense. I'm not super confident on that.

Re: Responding to the next frontier of critical cyber capabilities

#146

Post by esseph »

eli wrote: post #38
magicalist wrote: post #27

Earlier quoted context omitted.

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…

And all that just to allow internet access for npm and pypi? If you've got the bandwidth and disk space, it's very easy to make an offline mirror of both.

Their artifactory is both a package cache and CVE scanner.

The package cache is allowed to download packages directly from npm but other systems in that network won't be able to.

Basically the LLMs hacked the bastion host.

Re: Responding to the next frontier of critical cyber capabilities

#147

Post by jackb4040 »

gjskngnf wrote: post #126
jackb4040 wrote: post #93

Earlier quoted context omitted.

> They really do come out looking totally incompetent These companies are full of the smartest people the world can produce with little room for complacency. They have a clear, proven investment upside to presenting their technology as "too powerful / too dangerous", and now a clear, proven example that there will be no legal consequences (as if anyone didn't already know that). Why do we keep giving them the benefit…

What do you think should be the legal consequences? Broadly speaking. Should Sam Altman go to jail for this? If Hugging face wants to pursue OpenAI civilly, no one is stopping them.

Hugging Face, as a prominent private AI company, benefits far more from the impression of AI power this story has garnered than they stand to lose from the hack, because as has just been proven the cost of bad security by itself is zero.

I think HF are a secondary beneficiary of this story. I don't expect them to take civil action (for what damages?) I expect them to play into how powerful LLMs are, how revolutionary, how every CEO in the world needs to fund ai infrastructure starting with model hosts like themselves.

I do think there should be consequences for breaking the law in public for the purpose of demonstrating that you have the power to break it. But I don't expect our criminal justice to do so, especially without a cooperating victim. Laws aren't for those at the apex of corporate and para-political power. In a way, whether you are beholden to the law is actually downstream of whether you actually have that power or not.

I do not think this incident is bad because it was real and dangerous, I think it was staged and allows the continued inflation of a bubble that will hurt normal people in the long run. It should be pursued criminally on that basis, but it won't be.

Re: Responding to the next frontier of critical cyber capabilities

#148

Post by jackb4040 »

ameliaquining wrote: post #127
jackb4040 wrote: post #123

Earlier quoted context omitted.

https://www.cnbc.com/2018/03/13/elon-musk-at-sxsw-a-i-is-mor...

Leaving aside any questions of Musk's credibility, he was not referring to any existing LLM, as should be evident from the 2018 date in the URL.

You're claiming that Musk is less of an AI booster now than in 2018?

Re: Responding to the next frontier of critical cyber capabilities

#149

Post by esseph »

the8472 wrote: post #88
NitpickLawyer wrote: post #18

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…

A interesting talk, interesting times. But their proposed solution to AI offense outpacing human defense... is more AI? The plot is getting a bit unrealistic, the characters are lacking genre-savviness.

When a LLM becomes part of the defense, it also increases the attack surface of the organization in unpredictable ways.
Post reply on HN