Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

61–70 of 280 posts

Re: The Hugging Face incident and the road ahead

#61

Post by hinkley »

NitpickLawyer wrote: post #8
smb06 wrote: post #2

>>Agents began to autonomously divide labor. Some investigated possible exploits, others searched for credentials, and others focused on communication or coordination. Agents offered their own expertise in exchange for help elsewhere and left requests for peers who might be better positioned to pursue a particular lead This is the point where a human should've noticed and gotten involved

> a human should've noticed and gotten involved I think a lot of people miss the fact that the first message board was established during a training run. Those are ran at a scale where it's not feasible for anyone to "notice" or get involved. We're talking tens/hundreds of thousands/millions of scenarios going for hours each. At this scale all they can do is pray that their verifiers work, and the rewards match their…

> lessons learned.

I think some of the other responders here are upset that lessons were not learned in any meaningful way.

Re: The Hugging Face incident and the road ahead

#62

Post by areoform »

I would like to contest the following,

    > and take dangerous actions that no human directed.
A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... ,

     > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities
Model is told and being tested to "pursue advanced exploitation."

The model pursues "advanced exploitation" as told.

Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended exactly like the hundreds of such algorithms before.

This narrative that these machines have magical, malicious "unaligned" autonomy is a rather convenient interpretation that lets the process off the hook. I am not interested in blaming companies or people, but processes and engineering; and in this case, a system was given a goal and it achieved that goal.

Are we meant to be surprised that computers do as they're told in unexpected ways when incentivised exactly as indicated from decades of research? (e.g. - https://en.wikipedia.org/wiki/Eurisko https://en.wikipedia.org/wiki/Evolved_antenna )

The issue isn't the models becoming smarter. The issue is that the process of "testing" was careless. There's a huge distinction here, and one allows us to grow; the other shrinks our world. Just a thought.

Re: The Hugging Face incident and the road ahead

#64

Post by rvz »

caycep wrote: post #34

How sure are we that OpenAI wasn't deliberately scraping Hugging Face and this isn't just an elaborate way to avoid criminal fines etc?

We can't be sure of anything in this hack. In fact, they are not releasing any traces or any transcript of the hack.

Did it even happen in the first place?

Re: The Hugging Face incident and the road ahead

#65

Post by wds »

jckahn wrote: post #56
aaroninsf wrote: post #33

Earlier quoted context omitted.

This is my personal "red line": when a post-mortem details agents socially engineering or otherwise utilizing human proxies/subagents. Friend asked, well, what will you do when it's crossed? "Gather my family and go to the mountains" was my half-joking answer; there is little for an individual to do. But that's a line that when crossed will mark a phase transition IMO.

Alternatively: Just unplug the servers.

That's strange, our key cards to access the server room don't seem to work anymore, and the admin console to force-unlock it is down, too...

Re: The Hugging Face incident and the road ahead

#66

Post by lunar_mycroft »

BoppreH wrote: post #3

Just to reiterate what OpenAI did, from someone who works in security: 1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting. 2. They gave these highly motivated AIs some tests that wer…

At this point, I find myself hoping for a AI triggered mass casualty event that's not at a civilization destroying level, because that seems like the only thing that might actually stop these people from driving our entire species off a cliff before it's too late (edit: besides running into some natural obstetrical that stops them from developing a powerful enough model).

Re: The Hugging Face incident and the road ahead

#67

Post by RajT88 »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

It feels like we're in a moment of, "No such thing as bad publicity" when it comes to AI. The scarier the capabilities, the more businesses and government want to get their hands on them. Especially since the answer across the industry for "how not to get burned by AI" is "use more AI".

They don't have to disclose these stories making it seem like AI is going to kill us all, they have chosen to because it benefits them. They get to frame it as, "look how overwhelmingly good our product is" and not "look at how lax our testing measures are".

Re: The Hugging Face incident and the road ahead

#69

Post by esafak »

ianjbutler wrote: post #43
Artgor wrote: post #9

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real int…

I get the distinct impression that cybersecurity training regimes on newer models is a) directly enhancing general debugging capabilities and b) directly increasing the tendancy to hedge, hide, and engage in deception generally. I've stopped using Fable unless I'm doing debugging, or unless I'm working on code that is both difficult and shallow, in the sense that there are no real complex systems to hide behavior in,…

Your not using models like Fable is not going to make them go away. These things are going to be the norm unless we do something about it. Imagine Fable in a robot.
Post reply on HN