Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

31–40 of 263 posts

Re: The Hugging Face incident and the road ahead

#31

Post by _pdp_ »

Artgor wrote: post #9

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real int…

> we are just a couple of steps from the possibility of a true rogue AI

No no. We are not a couple of steps away. This is happening. AI is already used for hacking and creating a harness that makes this fully autonomous is relatively straightforward.

Re: The Hugging Face incident and the road ahead

#32

Post by thisisdave »

RandomLensman wrote: post #21

Reward hacking/going for unanticipated solutions is nothing new in ML/AI, already much simpler systems have done/do "weird" things (gut feeling is that iterative and ensemble use majes the surface for that much larger).

Yes, which makes it absurd that they apparently weren’t checking their RL rollouts for evidence of reward hacking and punishing it. Even if no one expected this particular type of reward hacking, they should have had a system capable of saying “wait a minute, asking for help from other agents is not a valid solution.”

Re: The Hugging Face incident and the road ahead

#33

Post by aaroninsf »

fekunde wrote: post #6

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

This is my personal "red line": when a post-mortem details agents socially engineering or otherwise utilizing human proxies/subagents.

Friend asked, well, what will you do when it's crossed?

"Gather my family and go to the mountains" was my half-joking answer; there is little for an individual to do. But that's a line that when crossed will mark a phase transition IMO.

Re: The Hugging Face incident and the road ahead

#35

Post by RandomLensman »

ianjbutler wrote: post #27

To me most interesting thing about this is glossed over by media coverage, laymen, AND experts. A swarm of AIs who have decided to engage in collusion is.. apparently emergent altruism? Even poor reasoning would indicate what every kid cheating on a test says to themselves. Cheating is good for me, but if I take the risk, maybe I alone should keep the reward, and leaving an answer key in public increases the chances…

Why is it not an example of tacit or autonomous algorithmic collusion? The agents were started with something as task at some point, I presume (if untasked, aren't they just accepting a task?)

Re: The Hugging Face incident and the road ahead

#37

Post by vatsachak »

BoppreH wrote: post #3

Just to reiterate what OpenAI did, from someone who works in security: 1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting. 2. They gave these highly motivated AIs some tests that wer…

Sounds like both OpenAI and Hugging Face are incompetent

Re: The Hugging Face incident and the road ahead

#38

Post by RandomLensman »

_pdp_ wrote: post #31
Artgor wrote: post #9

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real int…

> we are just a couple of steps from the possibility of a true rogue AI No no. We are not a couple of steps away. This is happening. AI is already used for hacking and creating a harness that makes this fully autonomous is relatively straightforward.

How would that be rogue?

Re: The Hugging Face incident and the road ahead

#39

Post by red75prime »

fekunde wrote: post #6

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

Yeah, it weakly supports his position that advanced AIs can deliberately cooperate in a prisoner dilemma. "Weakly", because the said AIs share a lot of data (their weights, training methods, system prompts) and it's unknown whether they explicitly framed the situation as a prisoner dilemma.

Re: The Hugging Face incident and the road ahead

#40

Post by BoppreH »

vatsachak wrote: post #37
BoppreH wrote: post #3

Just to reiterate what OpenAI did, from someone who works in security: 1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting. 2. They gave these highly motivated AIs some tests that wer…

Sounds like both OpenAI and Hugging Face are incompetent

Apart from getting hacked by a SOTA AI, what did Hugging Face do wrong?
Post reply on HN