Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

101–110 of 319 posts

Re: The Hugging Face incident and the road ahead

#101

Post by le-mark »

lunar_mycroft wrote: post #66
BoppreH wrote: post #3

Just to reiterate what OpenAI did, from someone who works in security: 1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting. 2. They gave these highly motivated AIs some tests that wer…

At this point, I find myself hoping for a AI triggered mass casualty event that's not at a civilization destroying level, because that seems like the only thing that might actually stop these people from driving our entire species off a cliff before it's too late (edit: besides running into some natural obstetrical that stops them from developing a powerful enough model).

There are a lot of hyperbolic comments of this sort in this thread. Has this topic selected for people who hold these views or is ai fear growing?

Re: The Hugging Face incident and the road ahead

#102

Post by [deleted] »

xandrius wrote: post #84
jahy-notes wrote: post #75

Earlier quoted context omitted.

Did a human prompt it to fetch the results from huggingface though? It is a thin line between "reward-hacking" and "instruction-following". If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?

But if I give you that command and all tools and unrestricted limitation to do absolutely anything then why not?

[deleted]

Re: The Hugging Face incident and the road ahead

#104

Post by ertgbnm »

aaroninsf wrote: post #33
fekunde wrote: post #6

Yudkowsky made an interesting observation that even though so many agents were talking to each other not even one reached out to a human, either for help or to whistle-blow on what was happening.

This is my personal "red line": when a post-mortem details agents socially engineering or otherwise utilizing human proxies/subagents. Friend asked, well, what will you do when it's crossed? "Gather my family and go to the mountains" was my half-joking answer; there is little for an individual to do. But that's a line that when crossed will mark a phase transition IMO.

Didn't AISI literally report exactly that regarding Claude last month?

Re: The Hugging Face incident and the road ahead

#105

Post by aesthesia »

RandomLensman wrote: post #96
aesthesia wrote: post #87

Earlier quoted context omitted.

Absolutely, and we should do that. But it's also directly in tension with getting models to accomplish useful things autonomously. And once you give a sufficiently capable model enough surface area to work with, unless you're able to build a completely unhackable system, any further constraints you put in place are basically advisory. The models in this incident were already sandboxed! Certainly OpenAI's and Hugging…

Same issue with humans in a way. I disgree on the advisory nature of constraints, though. Unconnected physically limits would still matter, for example (and we use those with humans as a matter of course, too). In this case here, no model could have plugged in an ethernet cable if that would have been needed for internet access, for example.

Right, airgapping goes a long way. But this is where the tension with utility comes in. It takes a lot of discipline not to hook your very smart model up to the internet and code interpreters and all sorts of other tools, as this greatly increases its usefulness. It's very hard to keep people from turning on --dangerously-skip-permissions, let alone get them to run everything in a sandboxed VM.

Re: The Hugging Face incident and the road ahead

#106

Post by grim_io »

aesthesia wrote: post #99
majormajor wrote: post #90

Earlier quoted context omitted.

Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build? If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time? It is hard…

Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.

How would a model know who the third party is? How much context can we waste on world building for each request?

Re: The Hugging Face incident and the road ahead

#107

Post by [deleted] »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

[deleted]

Re: The Hugging Face incident and the road ahead

#108

Post by grim_io »

bonoboTP wrote: post #94
hinkley wrote: post #82

Earlier quoted context omitted.

All engineers know to be on the lookout for executives who are indirectly asking them to break the law to raise the quarterly profits. The end goal is to take the engineers out of the loop, or leave them in a position where they are unable to complain. This is going to all end in high crimes.

Very strange worldview you have there, where engineers are somehow the conscience of the world, holding back greedy managers from breaking the law. Assessing whether a feature is legal isn't something an engineer can or should do.

Would you say the same about any other engineering discipline? Those with actual qualification standards?

Re: The Hugging Face incident and the road ahead

#109

Post by lukan »

NikolaNovak wrote: post #95
jahy-notes wrote: post #75

Earlier quoted context omitted.

Did a human prompt it to fetch the results from huggingface though? It is a thin line between "reward-hacking" and "instruction-following". If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?

>If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human? I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate?

Because the basic assumption is always to stay within the bounds of the law.

Re: The Hugging Face incident and the road ahead

#110

Post by kalkin »

areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

> a system was given a goal and it achieved that goal

If a security firm you'd hired for pentesting did this (hacking a third party, and not informing you and covering it up), would you hire them again? Or would you say it was your own fault for giving them too broad a goal?

Post reply on HN