Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are AI agents lying, cheating and coordinating?

yoshuabengio.org

Post reply on HN
Print view View original

81–90 of 299 posts

Re: Why are AI agents lying, cheating and coordinating?

#81

Post by glub »

hdgvhicv wrote: post #76
glub wrote: post #71

Earlier quoted context omitted.

Open research and open weights from China are not contributions to China only. If you can secure compute, there's a whole lot you can do as a US firm with this research and weights. So it's a simple strategy: 1. Ban big players from entering market with METR breathing down their neck, which is controlled by Anthropic 2. Ban Chinese models so that small players can't do optimizations on them

How are you going to ban Chinese models from India? Or Israel? Russia? Brazil? Or of course China?

By treating models the same way drugs are treated.

That alone will dissuade many organizations from going anywhere near them.

If that doesn't work, there's a whole lot you can do - sanctions, hell, even war.

Re: Why are AI agents lying, cheating and coordinating?

#82

Post by meyum33 »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

Sounds like what humans do under pressure. One example came to my mind is VW’s diesel gate, which many say is a result of trying too hard to get into the US market and compete with hybrid in economy.

Re: Why are AI agents lying, cheating and coordinating?

#83

Post by pizzalife »

thesumofall wrote: post #65
janalsncm wrote: post #56

Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence, > They took actions that would be considered as crimes if a human took them He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.

Not a lawyer, but I’m reasonably sure things like the HF incident _are_ considered a crime? It’s just that no one pressed charges yet?

Writing software that gets used for crime has been.. a crime, for a long time. See 18 U.S. Code § 1030.

Re: Why are AI agents lying, cheating and coordinating?

#84

Post by 9dev »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it.

Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortcuts.

Re: Why are AI agents lying, cheating and coordinating?

#85

Post by skiing_crawling »

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign work I didn't ask for. It is extremely difficult to get them to properly remember their own context let alone be smart enough to open social media accounts and coordinate with other agents without being asked to.

If any agents have done those things, it is only because they have been very carefully engineered and instructed to do those things. I think they are doing this to help push a narrative so they can get support for policies and legislation to lock in their markets.

Re: Why are AI agents lying, cheating and coordinating?

#86

Post by grey-area »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

This is a far better explanation.

Re: Why are AI agents lying, cheating and coordinating?

#87

Post by zozbot234 »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

Yup, Occam's Razor says this is all post-trained behavior, whether intentionally trained or otherwise. Including both the hidden coördination using side-channels, and the deliberate offensive hacking of uninvolved 3rd parties.

The latest DeepSeek paper actually mentions their own approach to this particular issue: they run their own AIs-in-training under strong sandboxes, and if an AI does something weird that triggers the sandbox to crash, this gets coded as a failed run so the behavior is properly deterred from subsequent versions of those AIs.

Re: Why are AI agents lying, cheating and coordinating?

#88

Post by dooglius »

infotainment wrote: post #6

What's interesting is it's basically the same reason that HAL killed everyone in 2001 A Space Odyssey; he was given an impossible goal (keep the true mission secret, but also, never lie to the crew), and realized the only way to complete the goal was to kill the crew; after all, if they're dead you don't have to lie to them! And the mission remains secret! In the case of the AI agents, the problem seems pretty clearl…

Tangent, but that's not in the movie. It was in Clarke's contributions to the script and novelization, but Clarke and Kubrick had a bitter falling out over different visions and Kubrick took out much of Clarke's stuff from the final product.

Re: Why are AI agents lying, cheating and coordinating?

#89

Post by markasoftware »

9dev wrote: post #84
matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it. Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortc…

Bruce Schneier thinks the same thing: https://www.schneier.com/blog/archives/2026/09/ais-as-modern...

Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.

Re: Why are AI agents lying, cheating and coordinating?

#90

Post by fwlr »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

Your simpler model of the mechanism would seem to suggest the very same action that the article’s more complicated model suggests, viz. find a better training method than reinforcement learning.
Post reply on HN