Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

361–370 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#361

Post by pixl97 »

JumpCrisscross wrote: post #332
zaptrem wrote: post #312

Earlier quoted context omitted.

Can you explain how the above event doesn't count as evidence alignment is an actual risk?

> Can you explain how the above event doesn't count as evidence alignment is an actual risk? Conflict of interest. Lack of a credible response. And no evidence of non-aligment. OpenAI and Hugging Face benefit from the Altman-Amodei catatrophy playbook, at least in the short term. If they believed this were a serious issue, the words air gap or law enforcement would have appeared in this post. And if "the models were…

There is plenty of evidence of things like inner misalignment. Things like this have always been issues in ML algorithms. At this point, you, and a large number of other people just wholesale throw out anything that isn't full speed ahead do whatever you want.

Are LLMs at the point of world wide catastrophe yet? No, I don't think so. Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it.

Re: OpenAI and Hugging Face address security incident during model evaluation

#362

Post by nrmitchi »

If you are attempting to run exercises like this, it is wildly negligent to not be running it in a physically-airgapped environment (potentially with a physical power shutdown).

You can not tell me that OpenAI doesn’t have the resources or ability to run tests like this in a physically-non-networked environment w/ sufficient compute for its needs.

Re: OpenAI and Hugging Face address security incident during model evaluation

#364

Post by pixl97 »

simoncion wrote: post #294
JumpCrisscross wrote: post #191

Earlier quoted context omitted.

> Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? Because we continue to have zero evidence that aligment is an actual risk.

> Because we continue to have zero evidence that aligment is an actual risk. I disagree. Every time one of these LLMs -say- interprets an attacker's instructions as either its system instructions or those of its user, interprets its own internal chatter as a user's command to perform a destructive operation on that user's data [0], burns all of the user's budget from getting stuck in an incredibly stupid loop, massiv…

Thank you, the "LLMs can do no wrong" bunch is ab exceptionally odd take from my point of view. LLMs are already causing all kinds of social issues, and the evidence of this exists in massive amounts. At least to me living in the US and the sue happy culture we have here, how much said AI providers have gotten away with so far surprises me.

Re: OpenAI and Hugging Face address security incident during model evaluation

#365

Post by JaRail »

gwd wrote: post #356
tdavies-dev wrote: post #72

Each time Anthropic would do their nonsense to get headlines about how theoretically dangerous their models were - like when they claimed a model blackmailed someone with emails showing he was cheating, but they basically pushed it as much as possible to do as such - it got me more and more worried. Because eventually it's going to be a boy-who-cried-wolf situation where scary stuff really does start happening but pe…

> Exploiting multiple zero-day vulnerabilities autonomously to escape containment is pretty nuts and the first story of this kind that I've heard. But this also feels like bragging under the guise of transparency. I mean, does it have to be one or the other? Just because it's actually dangerous doesn't mean nobody in OpenAI considers it great PR. And just because there are people in OpenAI that consider it great PR d…

Has to be mixed. The model accomplished something truly impressive. We'll see how impressive when the zero-days are available look at. But OpenAI as an engineering company screwed up. The impressive part is mostly locked away from public access so I don't see a huge PR upside. The ugly part could bite them and the entire AI industry hard in terms of regulations. People will be citing this for years.

Re: OpenAI and Hugging Face address security incident during model evaluation

#366

Post by bhouston »

slashdave wrote: post #123
himata4113 wrote: post #31

Earlier quoted context omitted.

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit

I dunno. I wonder if Sol could break OpenAI's security.

I bet it has multiple times but as part of white hate security testing.

Re: OpenAI and Hugging Face address security incident during model evaluation

#367

Post by elictronic »

blovescoffee wrote: post #75
elictronic wrote: post #56

This sounds an awful lot like pretending you have AGI so you can drum up your stock price. When you have a couple hundred billion dollars on the line I have zero faith in the messenger.

Huggingface literally reported the outage separately and did not know who caused it at first.

My comment from 15 hours ago. "There being squeezed by their own stock pumping and SpaceX pretending to be an AI company is driving down the exit strategy. I’m guessing one starts going full Theranos and begins claiming full AGI or gets the US government to government cheese then hard. It’s going to be a few crazy months."

I guess AGI it is huh. It is a little to obvious at this point.

Re: OpenAI and Hugging Face address security incident during model evaluation

#368

Post by pixl97 »

joe_the_user wrote: post #263
JumpCrisscross wrote: post #191

Earlier quoted context omitted.

> Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? Because we continue to have zero evidence that aligment is an actual risk.

I'd say that AIs occasionally "going crazy" and calling for death to human is evidence that these things might "mis-align" on occasion. And I say that knowing that most of these events are just these thing parroting bad sci-fi plots (or posts by people worried about alignment). That's true but everything they do is "just parroting" right?

If AI is just parroting humans, then training them with all the bad things humans do doesn't seem like the best of ideas. At the same time they have to 'know' these things to avoid being tricked. Kind of the eating the apple and gaining the knowledge of good and evil parable.

Re: OpenAI and Hugging Face address security incident during model evaluation

#369

Post by bad_haircut72 »

ofjcihen wrote: post #226
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

I’m honestly impressed that they managed to screw this up somehow. Setting up defense in depth, gaps, logical blocking etc is a standard practice for malware sandboxing. The entire purpose is to prepare for what you can’t foresee. This isn’t a new practice and I agree that this makes me wonder if they’re fit for this kind of research.

did you read the post? The model found new Zero-days to bypass existing blocks. Thats the point. Do you still think you can build a containment facility, which is still physically connected to the internet (only firewalled off or whatever) and contain it, if it can discover new unknown vulnerabilities in your whole plan?

Re: OpenAI and Hugging Face address security incident during model evaluation

#370

Post by nrmitchi »

conradkay wrote: post #327
gr_norm wrote: post #84

Why is a machine running these sorts of hacking benchmarks not airgapped? That seems a basic precaution, if OpenAI believes what they're selling. I mean, stuff like this is done for CTFs played by humans, too, to rule out collateral damage; it's not some new concept. So this is either thorough incompetence by OpenAI, a marketing piece, or both.

"Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries." Sounds like they just misunderestimated the model

Sure, but there is a definition of “airgapped” and that is not it.
Post reply on HN