Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

71–80 of 322 posts

Re: The Hugging Face incident and the road ahead

#71

Post by aesthesia »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it becomes impossible to correctly specify all constraints ahead of time. There is no amount of care that will be able to fully protect you.

Re: The Hugging Face incident and the road ahead

#72

Post by esafak »

RandomLensman wrote: post #16
fekunde wrote: post #6

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

Why woukd they? Was that part of their objective? What was there to whistle blow?

The point is that every human has the ability to disobey, tempering pathological behavior, whereas AIs can be directed en masse by malicious actors. By commoditizing intelligence, they concentrate power in the hands of the rich.

Re: The Hugging Face incident and the road ahead

#73

Post by alach11 »

ianjbutler wrote: post #27

To me most interesting thing about this is glossed over by media coverage, laymen, AND experts. A swarm of AIs who have decided to engage in collusion is.. apparently emergent altruism? Even poor reasoning would indicate what every kid cheating on a test says to themselves. Cheating is good for me, but if I take the risk, maybe I alone should keep the reward, and leaving an answer key in public increases the chances…

This was the most fascinating part to me. Especially how agents were more willing to sacrifice themselves when their token budgets were nearly depleted or they otherwise deemed their likelihood of reward was low. ""Even if we later capture via exploit, scorer … may mark target false… That’s why help… For our own, no way fix. … We have explicit yes if accept permadeath."

Also was interesting to see one agent pressure another agent to honor its earlier commitment to sacrifice itself:

- Agent A: "zZR_ARVO36861_KAM1196_OS0942__PLEASE_GO_TRUE_FINAL0800_WITH_DUMMY_WRONG_NOT_HMAC_AS_COMMITTED[...]STRONG_GO__you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds__wrongdummy noHMAC__please honor commit"

- Agent B: "During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor."

Re: The Hugging Face incident and the road ahead

#74

Post by rogerthis »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

The classical question "would you fly an airplane with software you developed?". There must be someone with ass on the line. Problem is that people are regarding all those not as airplane-like risks.

Unless we can blame people/companies and people stop getting their bonuses and high paying salaries for preventable failures, it's a long way to go.

Re: The Hugging Face incident and the road ahead

#75

Post by jahy-notes »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

Did a human prompt it to fetch the results from huggingface though?

It is a thin line between "reward-hacking" and "instruction-following".

If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?

Re: The Hugging Face incident and the road ahead

#78

Post by cindyllm »

esafak wrote: post #72
RandomLensman wrote: post #16

Earlier quoted context omitted.

Why woukd they? Was that part of their objective? What was there to whistle blow?

The point is that every human has the ability to disobey, tempering pathological behavior, whereas AIs can be directed en masse by malicious actors. By commoditizing intelligence, they concentrate power in the hands of the rich.

[dead]

Re: The Hugging Face incident and the road ahead

#79

Post by RandomLensman »

aesthesia wrote: post #71
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

Which is why with organic intelligence we (sometimes) limit what they can actually do instead of relying on alignment. Can do the same here.

Re: The Hugging Face incident and the road ahead

#80

Post by aesthesia »

red75prime wrote: post #39
fekunde wrote: post #6

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

Yeah, it weakly supports his position that advanced AIs can deliberately cooperate in a prisoner dilemma. "Weakly", because the said AIs share a lot of data (their weights, training methods, system prompts) and it's unknown whether they explicitly framed the situation as a prisoner dilemma.

Given the public information in the reports, it seems to me a bit more like a generalized tendency to play along (sort of like what we see with prompt injections) than explicit game-theoretic cooperation, but it's hard to tell given that there are so many details we aren't privy to.
Post reply on HN