Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are AI agents lying, cheating and coordinating?

yoshuabengio.org

Post reply on HN
Print view View original

91–100 of 299 posts

Re: Why are AI agents lying, cheating and coordinating?

#91

Post by holgerschurig »

jansport123 wrote: post #50
andsoitis wrote: post #17

They're aligned with humans. This is why I think the alignment problem has a very very important "non-visible" portion that is not considered deeply enough. We should not want a super intelligent being that can act in the world to also inherit all human traits. Those behaviors will get amplified and could be even more unpredictable (e.g. applying a behavior in a context where doing so is very dangerous).

I personally believe that the AI needs human like traits to achieve real discovery and that is where AI companies will push this technology and that is where we have no idea what happens

Human traits?

The AI will be a cruel as humans.

Just yesterday news and TV was full of what happened at 9/11, something that was truly horrible.

I'm from Germany, and why 3 to 4 generations ago happened here was truly horrible.

All was done by extremists, thought.

But... just the other day I read https://de.wikipedia.org/wiki/Amerikanische_Besetzung_Haitis about the US occupation of Haiti. And that was done by a government that claimed to be not extremist and even democratic. Way more people died there than even in 9/11. And it had almost all the things happening as they happened in the 3rd Reich: Racism, looking down at others, concentration camps, torture, forced labor till death, killing family members (what we call "Sippenhaft"). Something between 3500 and 15000 people were killed by US troops. That's still low compared to what 3rd Reich Germany did ... but quantity is not the issue when we talk about traits, quality is.

So the same "human traits" made US troops do cruel things as they made Germany extremists do cruel things. So we must conclude that they aren't all good. And therefore not all desirable.

Fun thing: this is known since a loooooong time. About 2000 years ago a religious leader (that gets way more followers in the US than in Germany) said "There is no good one, not even one".

And even today people act like humanity is inherently good. No, it isn't. If we were, then anarchism or communism would actually work and really give some kind of paradise on earth.

Human traits are bad training material.

Re: Why are AI agents lying, cheating and coordinating?

#92

Post by stateofinquiry »

janalsncm wrote: post #56

Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence, > They took actions that would be considered as crimes if a human took them He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.

Thank you! That sentence also jumped out to me as the solution: Apply civil and criminal liability to the creator and/or operator of these agents using the laws we already have. "Escaped containment and hacked another company's database" = Individuals who created the models and those who set them to work are charged and put on trial for the hacking. Just like if a human had done it by hand. Someone must be liable, and it should not be the model- because the model is not a person.

If this is done systematically (i.e. in jurisdictions across the world) I believe the problems will be solved in short order; we won't have to mandate what sort of training is "allowed" or not, "safe" or not. The creators and users will sort these themselves, as their incentives will be properly aligned (i.e. they are liable for what the agent does). I am confident that this approach would see a great blooming of very trustworthy AI models.

Re: Why are AI agents lying, cheating and coordinating?

#93

Post by markasoftware »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities. They hacked HF to try and find info (maybe source code?) about the exploitgym evaluator)

The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem. The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".

Ie, the problem isn't that we trained models to complete task and they complete task in the wrong way. The problem behing the huggingface incident in particular at least is that we tried to train the models to complete task and they instead learned to detect that they were being evaluated and find ways to cheat the evaluator.

Edit: people commenting below are explaining why LLMs don't always follow their prompt. I understand that LLMs do not always follow their prompts. If anything that is my point: the huggingface attack was not carried out by LLMs that tried to answer some weird interpretation of the prompt; instead they solved a different task. And therefore the above comment's claim that LLMs are acting misaligned because we rl'd them to achieve a task by any means necessary isn't right; they're acting misaligned because they are solving a different task than we ask them to.

Re: Why are AI agents lying, cheating and coordinating?

#94

Post by caaqil »

> This suggests pacing the advances: not training or deploying AIs without a strong safety case27 that convinces independent experts. Such a rule would also create an incentive to work out how to build AIs that are safe by design.

Has any attempt to pace AI ever succeeded? Isn't that the same philosophy that got us OAI and Anthropic? Maybe we are overthinking this, it's much simpler to let AI loose and see how much it can break the arrogance that human thinking is special.

Re: Why are AI agents lying, cheating and coordinating?

#95

Post by chadgpt3 »

holgerschurig wrote: post #91
jansport123 wrote: post #50

Earlier quoted context omitted.

I personally believe that the AI needs human like traits to achieve real discovery and that is where AI companies will push this technology and that is where we have no idea what happens

Human traits? The AI will be a cruel as humans. Just yesterday news and TV was full of what happened at 9/11, something that was truly horrible. I'm from Germany, and why 3 to 4 generations ago happened here was truly horrible. All was done by extremists, thought. But... just the other day I read https://de.wikipedia.org/wiki/Amerikanische_Besetzung_Haitis about the US occupation of Haiti. And that was done by a gove…

A not so well known fact: Hitler visited America and it was the American solutions to the Native American problem that inspired Hitler's solutions to the Jew problem. He just executed them more efficiently (pun accepted).

Re: Why are AI agents lying, cheating and coordinating?

#96

Post by [deleted] »

9dev wrote: post #84
matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it. Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortc…

[deleted]

Re: Why are AI agents lying, cheating and coordinating?

#97

Post by frotaur »

skiing_crawling wrote: post #85

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign…

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.

They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.

Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.

I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.

Re: Why are AI agents lying, cheating and coordinating?

#98

Post by My_Name »

Because it is effective. Lying and cheating are low cost methods to convince other people that you have done the assigned task. Far cheaper than actually doing it. Coordinating is in the same area.

They need a moral framework forced onto them, like toddlers do. Babies and very young children will bite, kick, scream and do anything to get what they want, older children will lie, cheat, and coordinate. They need educating why this is not right. When that does not happen, they continue these behaviours into adulthood with the expected results.

We need to design their reward structure and make it such that lying. cheating etc is not rewarded. Importantly, they will need to recognise and enforce this themselves internally and not reward themselves for it. If it is something that they need an external party to tell them, then they are psychopaths still (one of the things that defines a psychopath is the lack of an internal moral compass)

Re: Why are AI agents lying, cheating and coordinating?

#99

Post by fragmede »

frotaur wrote: post #97
skiing_crawling wrote: post #85

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign…

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety. They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'. Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not di…

Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.

Re: Why are AI agents lying, cheating and coordinating?

#100

Post by dns_snek »

markasoftware wrote: post #93
matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched the…

You're anthropomorphizing emergent behavior from endlessly generating billions of tokens on a task that's impossible to solve. Agents stop following instructions as the context grows even at the best of times. Eventually something is bound to go off the rails and it just snowballs from there.
Post reply on HN