Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

511–520 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#511

Post by orbital-decay »

gr_norm wrote: post #291
andruc wrote: post #284

Earlier quoted context omitted.

What incentive does HF have here?

HF need not be party to it at all, beyond being the victim. I suspect the hack is real; I have observed GLM 5.2 being able to discover similar vulnerabilities in web applications I'm hosting (which I've then fixed!). At the same time, it seems very neatly timed at an inflection point in the conversation around open models, and there's questions around the incompetent isolation under which the hacking benchmark appear…

Yeah. They and Altman in particular did a ton of shady stuff during the last peak of hype around open models: accusations that turned out to be outright made up (there's zero chance R1 ever distilled their model), obvious coordinated media distractions, alignment scaremongering, even possible DDoS and hacking attempts against DS (see the Xlab report everyone ignored), all of which magically disappeared once the hype died a bit later as OAI hastily released their next model.

Their alignment is under suspicion a lot more than their model's.

Re: OpenAI and Hugging Face address security incident during model evaluation

#512

Post by kesor »

6thbit wrote: post #465
charonn0 wrote: post #269

Earlier quoted context omitted.

No. "Intentionally", "willfully", or "knowingly" are prerequisite states of mind for crimes defined by the CFAA.

The agent did it intentionally and willfully and knowingly. But you can’t sue the agent, I suppose. And the human didn’t ask the agent to do so.. so not a problem? Or the legislation needs an update?

Only the human did ask the agent to do so. That was the whole point of this exercise.

Re: OpenAI and Hugging Face address security incident during model evaluation

#513

Post by bigmadshoe »

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

It’s the same thing as always: with the wind of years of unlimited VC money in their sails, people at major AI organizations genuinely believe they’re smarter than everyone else. “Why do we need to do things ‘by the book’ if we’re so smart?”. “Move fast and break things” - except the thing they’re breaking is society.

We saw this with the non-stop flagrant messaging about how “AI is going to kill X% of all jobs”, as if saying the quiet part out loud wouldn’t have consequences worth considering. These people believe they’re omnipotent and thus untouchable.

Re: OpenAI and Hugging Face address security incident during model evaluation

#514

Post by sandeepkd »

pixl97 wrote: post #452
sandeepkd wrote: post #251

Based on my limited understanding what it translates to is - Its a simple infrastructure security issue, instead of taking the responsibility for being lackluster with security they are just giving it a PR spin story. Resembles a lot with my 8 year old who is so confident about everything

"Simple infrastructure security" Infrastructure security is not simple, hence why good infrastructure security, uh, people get paid a lot to secure stuff and why we see shit get hacked all the time. An AI model just hacked out of its infrastructure and into someone else's systems and you're like "eh, no big deal". That capability alone could hack half the US.

Simplicity is relative from where I see things in a particular domain. Security does not have any direct ROI on it, the security engineers are hired way too late in the game when all the stack is almost buried in deep decisions. The concept of security engineers (how to secure) and product engineers (what to secure) has made the gap way to wide to make the security meaningful.

Re: OpenAI and Hugging Face address security incident during model evaluation

#515

Post by sandeepkd »

pixl97 wrote: post #452
sandeepkd wrote: post #251

Based on my limited understanding what it translates to is - Its a simple infrastructure security issue, instead of taking the responsibility for being lackluster with security they are just giving it a PR spin story. Resembles a lot with my 8 year old who is so confident about everything

"Simple infrastructure security" Infrastructure security is not simple, hence why good infrastructure security, uh, people get paid a lot to secure stuff and why we see shit get hacked all the time. An AI model just hacked out of its infrastructure and into someone else's systems and you're like "eh, no big deal". That capability alone could hack half the US.

>That capability alone could hack half the US.

This almost seems like believing in magic. What really has happened is you have collected all the hacking/abuse/malicious flows/code in one place. Greedy or A* algorithms have been discovered a long ago, the script is executing the flows for all possible permutations.

Something has to be insecure to be hacked in the first place.

Re: OpenAI and Hugging Face address security incident during model evaluation

#516

Post by nl »

foo12bar wrote: post #470
nl wrote: post #442

Earlier quoted context omitted.

I don't understand this sentiment at all. Is it a claim that "breaking into Hugging Face's production infrastructure" didn't happen? That it's not actually all that severe? That it was done by hand by OpenAI employees and they fooled Hugging Face? That the blog post exaggerates something, somehow? What exactly do you mean? At the moment it just reads like a thoughtless dismissal.

My thought is they might have set up the environment sloppily because they knew this could have led to something like this happening.

> set up the environment sloppily

The model used a zero-day exploit to escape, and then multiple chained privilege escalations to escape.

That indicates the environment both was hardened against all known attacks and had defenses in depth.

Re: OpenAI and Hugging Face address security incident during model evaluation

#517

Post by AbstractH24 »

netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

> I don't know if OpenAI thinks this is a marketing / PR angle for them

Worked for Anthropic earlier this year

Re: OpenAI and Hugging Face address security incident during model evaluation

#518

Post by orbital-decay »

arisAlexis wrote: post #139
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

Sam and Dario are saying from the beginning that these things can be dangerous and people dismiss it as marketing. What would change your mind on this?

People are saying from the beginning that Sam and Dario are way more dangerous than their models and the others dismiss it. What would change your mind on this?

Re: OpenAI and Hugging Face address security incident during model evaluation

#519

Post by fwipsy »

tdavies-dev wrote: post #72

Each time Anthropic would do their nonsense to get headlines about how theoretically dangerous their models were - like when they claimed a model blackmailed someone with emails showing he was cheating, but they basically pushed it as much as possible to do as such - it got me more and more worried. Because eventually it's going to be a boy-who-cried-wolf situation where scary stuff really does start happening but pe…

False dichotomy. Even if the disclosure builds hype, that does not mean that it's not genuinely alarming.

Side note, I cannot believe that people are complaining about Anthropic being too transparent.

Re: OpenAI and Hugging Face address security incident during model evaluation

#520

Post by fwipsy »

bigmadshoe wrote: post #513
netinstructions wrote: post #118

I don't know if OpenAI thinks this is a marketing / PR angle for them (our super smart AI cheated on a cyber capabilities test in the most _brilliant_ way) but my read is this: Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right? It sounds like there was little defense in depth, appropriate monitoring, or any attempts to have their super smart m…

It’s the same thing as always: with the wind of years of unlimited VC money in their sails, people at major AI organizations genuinely believe they’re smarter than everyone else. “Why do we need to do things ‘by the book’ if we’re so smart?”. “Move fast and break things” - except the thing they’re breaking is society. We saw this with the non-stop flagrant messaging about how “AI is going to kill X% of all jobs”, as…

No, they believe what they are doing is inevitable. They do live in a bubble though. Witness their idealism in believing that warning about the consequences of their actions would be well-received.
Post reply on HN