Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hugging Face incident and the road ahead

openai.com

Post reply on HN
Print view View original

91–100 of 318 posts

Re: The Hugging Face incident and the road ahead

#91

Post by nick__m »

nater5000 wrote: post #19
Artgor wrote: post #9

You know, it feels to me that we are just a couple of steps from the possibility of a true rogue AI. What would a rogue AI mean? AI that isn't controlled by humans. Technically, it is possible - if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: - intent (as I don't want to go into the talk about consciousness) - AI doesn't have real int…

Don't forget: there are plenty of humans that would love to help AI agents cause chaos, many of which would do so merely for the "lols," but also adversary governments, terrorist organizations, etc., would definitely appreciate the opportunity to support a rogue AI to cause whatever problems it can. So it's not just the risk of an AI managing to do this by itself (which is pretty risky in itself), but also the risk o…

They just have to find someone who believes in Rocko's basilisk, that makes an even better servant than someone who just want chaos.

Re: The Hugging Face incident and the road ahead

#92

Post by esafak »

RandomLensman wrote: post #83
esafak wrote: post #72

Earlier quoted context omitted.

The point is that every human has the ability to disobey, tempering pathological behavior, whereas AIs can be directed en masse by malicious actors. By commoditizing intelligence, they concentrate power in the hands of the rich.

Humans can and have been directed en masses by (what I would consider) malicious actors, too. The issue isn't new.

That goes without saying, but humans have the ability to ignore instructions, and they regularly do. There is only one instance of each model, and only a handful of them (that count, anyway).

Re: The Hugging Face incident and the road ahead

#93

Post by strange_quark »

doginasuit wrote: post #85
RajT88 wrote: post #67

Earlier quoted context omitted.

It feels like we're in a moment of, "No such thing as bad publicity" when it comes to AI. The scarier the capabilities, the more businesses and government want to get their hands on them. Especially since the answer across the industry for "how not to get burned by AI" is "use more AI". They don't have to disclose these stories making it seem like AI is going to kill us all, they have chosen to because it benefits th…

> It feels like we're in a moment of, "No such thing as bad publicity" It seems likely that's how the marketing at the frontier labs initially read the moment, but I don't think it is that moment. It is an open question how much regulation is warranted and there seems to be a very strong sentiment from the public and legislators that it should be significant.

The big bet is that the regulations are going to be so onerous that it pulls up the ladder from anyone other than the well-funded players. It's classic regulatory capture. They aren't very subtle about this, it's the whole point of their fear mongering and "but China" messaging.

Re: The Hugging Face incident and the road ahead

#94

Post by bonoboTP »

hinkley wrote: post #82
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

All engineers know to be on the lookout for executives who are indirectly asking them to break the law to raise the quarterly profits. The end goal is to take the engineers out of the loop, or leave them in a position where they are unable to complain. This is going to all end in high crimes.

Very strange worldview you have there, where engineers are somehow the conscience of the world, holding back greedy managers from breaking the law. Assessing whether a feature is legal isn't something an engineer can or should do.

Re: The Hugging Face incident and the road ahead

#95

Post by NikolaNovak »

jahy-notes wrote: post #75
areoform wrote: post #62

I would like to contest the following, > and take dangerous actions that no human directed. A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... , > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities Model is told…

Did a human prompt it to fetch the results from huggingface though? It is a thin line between "reward-hacking" and "instruction-following". If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?

>If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?

I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate?

Re: The Hugging Face incident and the road ahead

#96

Post by RandomLensman »

aesthesia wrote: post #87
RandomLensman wrote: post #79

Earlier quoted context omitted.

Which is why with organic intelligence we (sometimes) limit what they can actually do instead of relying on alignment. Can do the same here.

Absolutely, and we should do that. But it's also directly in tension with getting models to accomplish useful things autonomously. And once you give a sufficiently capable model enough surface area to work with, unless you're able to build a completely unhackable system, any further constraints you put in place are basically advisory. The models in this incident were already sandboxed! Certainly OpenAI's and Hugging…

Same issue with humans in a way. I disgree on the advisory nature of constraints, though. Unconnected physically limits would still matter, for example (and we use those with humans as a matter of course, too). In this case here, no model could have plugged in an ethernet cable if that would have been needed for internet access, for example.

Re: The Hugging Face incident and the road ahead

#98

Post by sscaryterry »

bonoboTP wrote: post #94
hinkley wrote: post #82

Earlier quoted context omitted.

All engineers know to be on the lookout for executives who are indirectly asking them to break the law to raise the quarterly profits. The end goal is to take the engineers out of the loop, or leave them in a position where they are unable to complain. This is going to all end in high crimes.

Very strange worldview you have there, where engineers are somehow the conscience of the world, holding back greedy managers from breaking the law. Assessing whether a feature is legal isn't something an engineer can or should do.

Hmm, engineers are expected to know what is legal and not.

Re: The Hugging Face incident and the road ahead

#99

Post by aesthesia »

majormajor wrote: post #90
aesthesia wrote: post #71

Earlier quoted context omitted.

This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it…

Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build? If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time? It is hard…

Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.

Re: The Hugging Face incident and the road ahead

#100

Post by RandomLensman »

esafak wrote: post #92
RandomLensman wrote: post #83

Earlier quoted context omitted.

Humans can and have been directed en masses by (what I would consider) malicious actors, too. The issue isn't new.

That goes without saying, but humans have the ability to ignore instructions, and they regularly do. There is only one instance of each model, and only a handful of them (that count, anyway).

But the people directing them are there. We have long experience with limiting people although it might sometimes not look like that so much.
Post reply on HN