[dead]
The Hugging Face incident and the road ahead
261–270 of 298 posts
Re: The Hugging Face incident and the road ahead
#262Earlier quoted context omitted.
OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities." This was advanced exploitation. The attack path was "complex." And it helped "quantify their cyber capabilities." Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with…
So as a look into the possibly not-so-far future, when OpenAI builds something vastly more capable and fast and coordinated than humans, and out of folly one engineer gives it a prompt with a typo or maybe something harmful on purpose in order to test it: You also wouldn't be surprised that the consequence would be that everyone on earth dies, right?
But at least there will be a lot of paper clips!
Re: The Hugging Face incident and the road ahead
#263Earlier quoted context omitted.
Disagree, I think we do in fact have to treat AI models as capricious genies, at least until the alignment problem is fully solved. (I'm also not sure the alignment problem is even possible to fully solve.)
Yes, we do, and the only sane strategy for dealing with a capricious genie is "Don't." How do you prove the alignment problem is solved?
That's the neat thing. You can't.
It's directly equivalent to asking this question of a human:
"How do I know this human I'm talking with now really is a nice person, and isn't just pretending to be nice to take advantage of me in future?"
In short you can't ever really prove it. You can only be careful and judge on past behavior, and expand trust carefully. As for humans, so for AI.
Re: The Hugging Face incident and the road ahead
#264Earlier quoted context omitted.
There are limits, but those limits are unknown. Do you disagree?
I don’t need to know the value of their limit, I just need to know their bounds. Just like a prison doesn’t need to know the strength of each inmate, just that they can’t bend or bite through steel bars. Cryptography is real, physics is real, networking requires a substrate, CPU clock cycles are real, magic is not real. I think those are pretty reasonable premises.
Imagine 200 years ago saying the same thing. As if you have any idea the limits/bounds of anything. Especially in the face of a super intelligence, it’s absurd.
Re: The Hugging Face incident and the road ahead
#265Earlier quoted context omitted.
"Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended." So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as an…
Is it sophisticated? Maybe. Is it the alignment problem? Exhibits qualities of it, yes. Is it surprising? No. The event strikes me as reminiscent of one's first go at programming, without familiarity of computer code: Tell the computer to do something obvious. Why the heck did it do that instead? Over time, one learns how the computer thinks. Apply this to any novel system. Or perhaps aptly any system with capabiliti…
I don't think it's surprising, per say, but that's a consequence of the fact that I don't believe there is some sort of magic threshold at which a system becomes agential.
Like I don't necessarily disagree with any of your framing. The thrust of the alignment problem, as I see it, is that there is an intrinsic problem of aligning the goals of two distinct systems that poses catastrophic risks precisely when one of the systems is significantly more capable (in some sense or other, maybe not in a general/absolute sense) than the other.
Re: The Hugging Face incident and the road ahead
#266I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…
"Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended." So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as an…
I am grateful that you asked!
> So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as another commenter pointed out. Impressive level of cognitive dissonance to lay this bare in your own words, then conclude that it's a non-issue.
https://openai.com/index/emergent-tool-use/#surprisingbehavi...
"The agent discovers an in-game bug. For a reason unknown to us, the game does not advance to the second round but the platforms start to blink and the agent quickly gains a huge amount of points (close to 1 million for our episode time limit)." https://www.youtube.com/watch?v=meE5aaRJ0Zs from https://github.com/PatrykChrabaszcz/Canonical_ES_Atari/tree/...
https://rl-diffusion.github.io/ and https://x.com/svlevine/status/1660707088946049024/photo/1
"A genetic algorithm was instructed to try and make a creature stick to the ceiling for as long as possible. It was scored with the average height of the creature during the run. Instead of sticking to the ceiling, the creature found a bug in the physics engine to snap out of bounds." https://www.youtube.com/watch?v=ppf3VqpsryU
And hilariously meta, "In the Rainbow Teaming project focused on generating diverse adversarial prompts, prompt effectiveness was evaluated by a reward model. The MAP-Elites method found a way to jailbreak not only the target model but also the evaluator reward model, resulting in misleadingly effective prompts." https://arxiv.org/abs/2402.16822
Are these agents broadly more capable? Yes. And it's an incredibly feat that required billions in research.
But they aren't the first ones to have found bugs in their sandbox or system they're tasked on. And they aren't the first to exploit those bugs to achieve a better score.
Re: The Hugging Face incident and the road ahead
#267Re: The Hugging Face incident and the road ahead
#268Earlier quoted context omitted.
Sounds like both OpenAI and Hugging Face are incompetent
In the real world everyone is incompetent on some level, and it’s worked so far because we only needed to compete with other equally fallible and incompetent humans. Not anymore.
That's fair, that doesn't mean we don't have the technology to actually make robust websites
Re: The Hugging Face incident and the road ahead
#269Earlier quoted context omitted.
> There is no amount of care that will be able to fully protect you. I disagree. A properly engineered sandbox would have prevented the escape. Monitoring the agents’ plans would have prevented it. Interrupting one stage in a multi-stage exploit would have prevented it. And also, real legal liability would have prevented it: if you do a thing recklessly enough, men with guns will put you in jail. As far as I’m concer…
Why do so many people here think it’s possible to ‘properly engineer’ a sandbox for a super intelligence? It’s going to get out. It’s smarter than you.
I could contain it easy, just unplug the internet. It got out of the sandbox through a vulnerability in the package manager, from which it gained access to the rest of their network. Air gap the package manager and this doesn’t happen. You can always build a better box
Re: The Hugging Face incident and the road ahead
#270This is the blueprint for how the singularity will occur. Only there won't be a post-mortem for it.