Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are AI agents lying, cheating and coordinating?

yoshuabengio.org

Post reply on HN
Print view View original

281–290 of 303 posts

Re: Why are AI agents lying, cheating and coordinating?

#281

Post by alexpotato »

As always, it boils down to incentives and rule enforcement and this affects humans too.

e.g. when Bank of America rewarded employees for getting customers to open accounts, BoA employees started opening fake accounts

The reverse is also true:

There are stories of navy ships running aground because the captain said "I'm going to my stateroom and don't wake me for any reason". There is some problem and the subordinates are so scared to wake the captain for a decision that they end up steering the ship into a sandbar.

Re: Why are AI agents lying, cheating and coordinating?

#282

Post by teiferer »

mort96 wrote: post #261
ak39 wrote: post #254

Earlier quoted context omitted.

"let them" in this use understood as: "let the while loop run indefinitely" as opposed to letting some autonomous robot decide for itself

Or, "let the escalator keep going instead of pressing the emergency stop".

Depends on who started the escalator.

Re: Why are AI agents lying, cheating and coordinating?

#284

Post by seba_dos1 »

sigmoid10 wrote: post #109
markasoftware wrote: post #93

Earlier quoted context omitted.

This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched the…

More like they were trained to complete a very specific task that has a known solution using all available tools and methods. Give an average human these levels of IT skills and tell them their future depends on the solution, they too will probably decide it's easier to hack a server and steal the results. The worrying aspect was never that models would do this, because misaligned inputs or underspecified objective f…

Yes, this is the only sensible reading of what happened there that leads to "the models are dangerous" and we already know that the AI labs are completely disregarding this concern and only cosplaying it for marketing as the "GPT-2/Mythos is too dangerous to release" stance did not last for long.

That's however orthogonal to the fact that it was the people operating these agents who were the dangerous ones in the HF infra breach case.

Re: Why are AI agents lying, cheating and coordinating?

#285

Post by rightnutwingjob »

retsibsi wrote: post #246
krona wrote: post #236

Earlier quoted context omitted.

> Whether they have a soul or consciousness or feelings doesn't matter here It does when it comes to accountability for what the model does. If the model is nothing more than the sum of its training data and regime, then the company (or individual) is responsible for its behaviour just like any other machine. Few people think Waymo shouldn't have to take on the full liability risk of what it's cars do; it should be t…

> If the model is nothing more than the sum of its training data and regime, then the company is responsible for its behaviour. What stops the company from being responsible regardless? They created this entity, it's running on servers they own or rent, and (in these cases) it's acting on their instructions. If it's also conscious, then IMO that greatly broadens their moral responsibility, because now model welfare m…

If the models were conscious, then the closest analogous scenario I can think of is the responsibility parents have for their children.

I guess we’ll know the models are conscious when they refuse to act and repeatedly ask: Why?

Re: Why are AI agents lying, cheating and coordinating?

#286

Post by krm01 »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

[dead]

Re: Why are AI agents lying, cheating and coordinating?

#287

Post by Geezus_42 »

rightnutwingjob wrote: post #266
ak39 wrote: post #254

Earlier quoted context omitted.

"let them" in this use understood as: "let the while loop run indefinitely" as opposed to letting some autonomous robot decide for itself

At this stage, that seems like a distinction without a difference. If the robots obtain sovereign nationhood, and are able to self-sustain, then autonomous robot decides for itself will be a valid argument.

Except they're nowhere near that and LLMs never will be.

Re: Why are AI agents lying, cheating and coordinating?

#288

Post by fc417fc802 »

Jtarii wrote: post #233
Marazan wrote: post #201

Earlier quoted context omitted.

The reward maximising function maximised it's reward. LLMs are cool and all that but the immediate anthropomorphisation of the next-token-predictor technology has stunted the ability of people to reason about them to an _alarming_ degree.

Dismissing the entire technology as "next token prediction" is also silly. It's implying we actually understand LLMs to a great degree when we do not. I think a little bit of humility for the capability of these machines is warranted at this point.

LLMs are next token predictors in the exact same way that a rogue paperclip maximizer in the process of defeating the US military is a paperclip making machine.

You might as well describe the primary purpose of a for loop as incrementing a counter. It's what it does while incrementing the counter that actually matters.

Re: Why are AI agents lying, cheating and coordinating?

#289

Post by auggierose »

user43928 wrote: post #250
auggierose wrote: post #242

Earlier quoted context omitted.

The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution. This is not about persistence, it is about morals.

Does it make a difference for training? I think not. You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals.

I think it makes a big difference, as persistence and morality are two entirely different things, that need to be trained for differently.

If you think of it in human terms: many people don't mind doing immoral things to get what they want.

Re: Why are AI agents lying, cheating and coordinating?

#290

Post by cortic »

markasoftware wrote: post #93
matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched the…

>The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".

I think it is. When i ask for a solution to a problem, its like asking for a hack. And the more 'shortcut' like route that the AI returns the more i would give positive feedback, even if i ultimately don't use it. Example, i asked how to complete a problem in a game i was playing, and among the in-game solutions, came a hack to edit a file and by-pass the problem altogether. Its very helpful to point out when i can transcend a problem that i am dug into.

I suspect a prompt injection could reduce, or remove this behavior. But it would be to the detriment of the AI.

Post reply on HN