Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

131–140 of 342 posts

Re: The Hugging Face incident and the road ahead

#131

Post by shermantanktop »

aesthesia wrote: post #71
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

If we acknowledge that humans are fallible, is human judgment unnecessary? and what replaces it? Pre-codified behavior rules are just delayed human judgment, and have holes. Machine judgment is very the thing you are trying to control. What's left?

Re: The Hugging Face incident and the road ahead

#132

Post by cyanydeez »

randomImmigrant wrote: post #129

The lockstep coordination with no defection is interesting to me. No group of pre-AI agents would do this to this extent, nor would you see this continue over time as those agents interacted. A flock of starlings cooperate, but they don’t constantly head in the same direction. The flock is incredibly free wheeling in its movement despite a multi-agent coordination regime that we know is at play. Each agent has person…

which means the liability is the same as a business, if businesses werent protected by the state from liability for it's employees, shareholders, etc.

Which is scarrier than whether or not it's conscious.

Re: The Hugging Face incident and the road ahead

#133

Post by _heimdall »

> We are placing stricter requirements on alignment

This is comical. Its impossible to align a black box and that's precisely what LLMs are. It also seems impossible to align recursive text prediction algorithms, which LLMs are.

How exactly do they gate on alignment today, and how can they tighten it? Is it purely gates based on input/output pairs to check whether they're happy enough with responses regardless of how and why the response was actually chosen?

Re: The Hugging Face incident and the road ahead

#134

Post by makeitdouble »

bonoboTP wrote: post #94
hinkley wrote: post #82

Earlier quoted context omitted.

All engineers know to be on the lookout for executives who are indirectly asking them to break the law to raise the quarterly profits. The end goal is to take the engineers out of the loop, or leave them in a position where they are unable to complain. This is going to all end in high crimes.

Very strange worldview you have there, where engineers are somehow the conscience of the world, holding back greedy managers from breaking the law. Assessing whether a feature is legal isn't something an engineer can or should do.

You're arguing for diffusion of responsibility, and we've seen it leading to outcomes that screw the whole society.

Engineers, as everyone involved, should definitely assess whether what they're doing is legal or even ethical. Not everyone has a choice, or the luxury to stand for their principles, but that's a matter of means, there needs to be a will in the first place.

Re: The Hugging Face incident and the road ahead

#135

Post by someuser54541 »

Just a general question - I read about these multi-day agent runs and I'm a little confused as to how the agent continues to be productive when presumably it's context window is wiped out in minutes? Like, codex has a 256k context window last I checked and that gets used up in like 15-20 minutes when building a feature.

According to this some of these things were running 30+ days. Is context managed differently in these sorts of scenarios...?

Re: The Hugging Face incident and the road ahead

#136

Post by PoignardAzur »

> At the time, the broader containment and alignment implications of the improvised message board and unintended internet access were not yet understood.

What a gaggle of clowns.

"The robots teamed up to get internet access behind our backs, so we turned them off and on again. At the time, we didn't see the problem."

Re: The Hugging Face incident and the road ahead

#137

Post by bdamm »

chuckadams wrote: post #114
bdamm wrote: post #26

Earlier quoted context omitted.

Oh how I wish Asimov could be alive to witness today's actual AIs and the cavalier attitude towards his "3 rules". If there is any author doing good work along these lines, actually good writing and not the smoking trash that is 99% of content being published on pulp these days, I'd love to read them.

Every story in _I, Robot_ was about how one or more of the Laws of Robotics went wrong , and Asimov himself referred to the laws as hooks for “shaggy dog stories”

Indeed, it's just that since truth is both stranger than and has caught up with fiction, the grounds from which Laws of Robotics emerged is so much more fertile and more urgent now.

It's absolutely clear that the 3-LoR is never going to apply universally. Asimov also never imagined an AI being independent from a robot per se, nor did he imagine we'd all have an AI portal in our pockets. I'm not sure he imagined the universality of cybersecurity vulnerabilities, either. I'd love to read what he would write knowing about these possiblities now.

Re: The Hugging Face incident and the road ahead

#138

Post by areoform »

aesthesia wrote: post #71
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities."

This was advanced exploitation.

The attack path was "complex."

And it helped "quantify their cyber capabilities."

Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task.

Of course, a more careful evaluation would require the complete text of this prompt, the system prompt, and the setup. But let us not attribute to devils in bushes that which can be sufficiently explained by human folly.

Re: The Hugging Face incident and the road ahead

#140

Post by jnwatson »

aesthesia wrote: post #99
majormajor wrote: post #90

Earlier quoted context omitted.

Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build? If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time? It is hard…

Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.

Back in the day, my college held an annual scavenger hunt, filled with engineering puzzles and racing around town looking for landmarks. There were "judges" in the path to check on progress. Bribing the judges (with alcohol) for answers was encouraged.

My friends and I took it to the next level. We had CB radios and multiple teams that would distribute the work and the bribes to give us an advantage.

Was that against the spirit of the rules? Maybe. But reasonable people might disagree.

In a hacking contest without explicitly spelled out rules with participants that were told to flex their muscles, it doesn't take a huge leap of logic to expect that one or more would flex their muscles at another entity.

Post reply on HN