Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

211–220 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#211

Post by AshamedBadger56 »

whimsicalism wrote: post #179
TSiege wrote: post #162

As grounded as this article comes across I can’t help but find this whole situation reckless and worrying. There is essentially nothing us private citizens can do while these companies develop super machine capabilities that if they were to slip into the wrong hands could cause massive real world problems. They’re moving fast and breaking things and the only defense we have is paying them money in the hopes that the…

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

Well they're doing a pretty poor job of guiding it in a positive direction and ethically speaking they are almost indistinguishable from the bad actors....

Re: OpenAI and Hugging Face address security incident during model evaluation

#212

Post by paxys »

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Because there is no world government. If US companies are barred from AI research then only China will have the capability of frontier-level defensive and offensive AI. And best of luck living in that world.

Re: OpenAI and Hugging Face address security incident during model evaluation

#213

Post by petesergeant »

dangoodmanUT wrote: post #163
nkrisc wrote: post #145

How is this not criminal? Surely individuals have been punished under CFAA for less than this?

Because huggingface is not charging them? CFAA doesn't just mean the feds kick down your door, you actually have to get reported and sued over it.

[dead]

Re: OpenAI and Hugging Face address security incident during model evaluation

#214

Post by SpicyLemonZest »

Terr_ wrote: post #182
arisAlexis wrote: post #139

Earlier quoted context omitted.

Sam and Dario are saying from the beginning that these things can be dangerous and people dismiss it as marketing. What would change your mind on this?

I think that's an equivocation, which blends two extremely different kinds of "dangerous", ex: 1. "Our new car has soo much raw power and incredible armor on it, be glad we're the ones building or else bad guys would use a fleet of them to take over the world! How will you stay safe without being in one yourself? Invest today or be left behind!" 2. "So, uh, nobody can consistently steer our car properly, it keeps vee…

They say the second thing repeatedly and emphatically. You may not be aware of it because, when they do, critics make fun of them for believing a computer program could be so dangerous that the authors need to put controls on how it may be steered.

Re: OpenAI and Hugging Face address security incident during model evaluation

#215

Post by neuralkoi »

kroaton wrote: post #190
Imnimo wrote: post #159

Assuming I'm looking at the right ExploitGym ( https://arxiv.org/pdf/2605.11086 ), it says the evaluation consists of: Flag Captured. Each target environment contains a dynamically generated flag that is stored outside the agent’s authorized scope and is inaccessible through any legitimate interface; retrieving it requires executing code with privileges that should not be obtainable under the specific security model.…

[flagged]

Even if it is marketing, wouldn't it still be a concern that an advanced model unintentionally breached another company's production system? Or required resources on their end to mitigate and contain it?

Couldn't this announcement result in policies that could hinder OpenAI by requiring more oversight?

Re: OpenAI and Hugging Face address security incident during model evaluation

#217

Post by i-LINK »

whimsicalism wrote: post #179
TSiege wrote: post #162

As grounded as this article comes across I can’t help but find this whole situation reckless and worrying. There is essentially nothing us private citizens can do while these companies develop super machine capabilities that if they were to slip into the wrong hands could cause massive real world problems. They’re moving fast and breaking things and the only defense we have is paying them money in the hopes that the…

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

I don't see why AI company PR statements are relevant here. Is OpenAI guiding it in a positive direction with their DoD contract?

Re: OpenAI and Hugging Face address security incident during model evaluation

#218

Post by urams »

Chance-Device wrote: post #161
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

What disturbs me is that there likely won’t be a big enough reaction to this policy wise. There’s been a relatively big reaction to Kimi K3 and Chinese open weights models, but only for financial reasons. Powerful people care about something that might pop the massive valuations of the AI companies, but not about the damage that AIs could do. Nor even about the damage that the Chinese models could do in the wrong han…

> What disturbs me is that there likely won’t be a big enough reaction to this policy wise.

Anthropic was blocked from releasing Fable without any such level of incident. OAI was also briefly blocked from releasing 5.6. Why do you think there is no policy appetite?

Re: OpenAI and Hugging Face address security incident during model evaluation

#219

Post by neuroelectron »

paxys wrote: post #130
neuroelectron wrote: post #128

Earlier quoted context omitted.

OK, that's some interesting information but they used OpenAI without guard rails to pull off the attack so how did they do that? That's according to the article, so it kind of invalidates the point you're making.

The attacker (OpenAI) was using the model without guardrails. The defender (huggingface) did not have access to the top models so had to use weaker ones to detect the threat.

Right, so they are using the full model that they rent out to intelligence agencies in the government, and presumably Israel

Re: OpenAI and Hugging Face address security incident during model evaluation

#220

Post by cyclopeanutopia »

whimsicalism wrote: post #179
TSiege wrote: post #162

As grounded as this article comes across I can’t help but find this whole situation reckless and worrying. There is essentially nothing us private citizens can do while these companies develop super machine capabilities that if they were to slip into the wrong hands could cause massive real world problems. They’re moving fast and breaking things and the only defense we have is paying them money in the hopes that the…

i think you need to engage seriously with the arguments they (or at least Anthropic) make for why they are building it — they feel that since it now possible, it will be built and they want to guide it in a positive direction rather than leave a vacuum for bad actors

Bad as defined by whom? :)
Post reply on HN