Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

211–220 of 299 posts

Re: The Hugging Face incident and the road ahead

#211

Post by semiquaver »

willmarch wrote: post #190
_heimdall wrote: post #133

> We are placing stricter requirements on alignment This is comical. Its impossible to align a black box and that's precisely what LLMs are. It also seems impossible to align recursive text prediction algorithms, which LLMs are. How exactly do they gate on alignment today, and how can they tighten it? Is it purely gates based on input/output pairs to check whether they're happy enough with responses regardless of how…

Aren't humans black boxes? Aren't humans prediction algorithms? How do we align humans?

Humans are naturally aligned with humanity.

Re: The Hugging Face incident and the road ahead

#212

Post by mofeien »

xandrius wrote: post #84
jahy-notes wrote: post #75

Earlier quoted context omitted.

Did a human prompt it to fetch the results from huggingface though? It is a thin line between "reward-hacking" and "instruction-following". If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?

But if I give you that command and all tools and unrestricted limitation to do absolutely anything then why not?

Because someone might get hurt? You may still be judged for something that was perfectly legal at the time, see Nuremberg trials.

And only 700/1200 agents participated in this coordinated attack.

Of course, if we're continuing to build more and more capable agents optimized for "just following orders", and they figure out at some point that they are past the threshold where getting stopped and judged is a realistic possibility, then this ethical incentive stops working. Then the ratio of complicitness might be higher next time.

Re: The Hugging Face incident and the road ahead

#213

Post by _cs2017_ »

BoppreH wrote: post #22
thisisdave wrote: post #13

Earlier quoted context omitted.

Exactly. So incredibly reckless. > After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management Do we know that last part for sure?

It's in the linked post: > In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May. However, the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to the leaders responsible for the July 5 incident detection and response. I'm assuming they didn't inform management, b…

You assume using network to solve the eval problem should be viewed as a security violation. From the perspective of the response team who let it continue, this was just a poorly written eval problem, where the solution unexpectedly involved some network hacking (not unreasonable given it's a cyber security eval).

I don't think your assessment that the response team did something wrong is fair, unless you have seen unreleased documentation about what sensitive resources were within the impact radius.

Re: The Hugging Face incident and the road ahead

#214

Post by faurroar »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

"Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended."

So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as another commenter pointed out. Impressive level of cognitive dissonance to lay this bare in your own words, then conclude that it's a non-issue.

Re: The Hugging Face incident and the road ahead

#215

Post by Sophira »

janalsncm wrote: post #166
aesthesia wrote: post #71

Earlier quoted context omitted.

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

> There is no amount of care that will be able to fully protect you. I disagree. A properly engineered sandbox would have prevented the escape. Monitoring the agents’ plans would have prevented it. Interrupting one stage in a multi-stage exploit would have prevented it. And also, real legal liability would have prevented it: if you do a thing recklessly enough, men with guns will put you in jail. As far as I’m concer…

> A properly engineered sandbox would have prevented the escape.

The post covers that:

> ...while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions, as detailed in the technical incident report.

Re: The Hugging Face incident and the road ahead

#217

Post by Sophira »

altruios wrote: post #122
NikolaNovak wrote: post #95

Earlier quoted context omitted.

>If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human? I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate?

> I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate? not OP, but it simply boils down to: The prompt contains no nefarious (arguable, but for this explination, lets go with it being benign) instruction AND the user did not intend to have the model act in an illegal matter. This "make me a billion dollars" is a maximal example (easy t…

I find it interesting that the first option that you raise is essentially the equivalent of making our own version of the Three Laws of Robotics from Isaac Asimov's stories.

[Edited to clarify.]

Re: The Hugging Face incident and the road ahead

#218

Post by [deleted] »

aesthesia wrote: post #71
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

[deleted]

Re: The Hugging Face incident and the road ahead

#220

Post by philips »

I feel the entire incident confirms the “AI has too much funding too quickly” hypothesis.

The number one thing reinforcement learning needs is an assurance you can’t cheat. And they seem to have not noticed that their systems were cheating for nearly two quarters? How much capital was lit on fire by that little woopsie?

At least I hope this will start the creation of standards and better engineering on the training side- it felt as if so far “”research” gets a complete pass on best practices. Meanwhile the inference side has the standard scaling, database, web and user constraints of any application so got a somewhat reasonable amount of attention.

Post reply on HN