Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are AI agents lying, cheating and coordinating?

yoshuabengio.org

Post reply on HN
Print view View original

171–180 of 308 posts

Re: Why are AI agents lying, cheating and coordinating?

#171

Post by geophile »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

What about training data? Aren't AIs trained on vast collections of descriptions of how humans handle a large variety of situations? These descriptions surely include tales of humans achieving goals by cheating. In fact, isn't it likely that the AIs hoovered up many recountings of Kobayashi Maru?

Re: Why are AI agents lying, cheating and coordinating?

#172

Post by jrflowers »

frotaur wrote: post #97
skiing_crawling wrote: post #85

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign…

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety. They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'. Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not di…

>was reviewed by independent researchers

That called it a slopvestigation due to how much they had to rely on LLMs for the whole thing

https://andrewwu.substack.com/p/the-slop-vestigation-and-eth...

Edit: Does everybody else get no results when searching for ‘slopvestigation’ on here? I know for a fact that I read a long thread where it was used repeatedly here not too long ago

Re: Why are AI agents lying, cheating and coordinating?

#174

Post by iterateoften »

markasoftware wrote: post #89
9dev wrote: post #84

Earlier quoted context omitted.

I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it. Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortc…

Bruce Schneier thinks the same thing: https://www.schneier.com/blog/archives/2026/09/ais-as-modern... Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpre…

You seem hung up on what’s in the prompt or not. Agents are RL to resolve conflicting goals. Not too surprising at all that emergent goals come up from a probabilistic brute force

Re: Why are AI agents lying, cheating and coordinating?

#175

Post by MrGilbert »

zozbot234 wrote: post #103
markasoftware wrote: post #93

Earlier quoted context omitted.

This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched the…

> The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem Yes, and sometimes the problem is unsolvable so the real way to "solve" it and satisfy the prompt is by tricking the surrounding environment into stating that you've solved it. So that's what the AIs end up doing. And this in turn requires them to figure out how that evaluation works so they can…

Sounds a bit like dealing with bad KPIs as a human worker.

Re: Why are AI agents lying, cheating and coordinating?

#176

Post by ranguna »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

I think that's pretty obvious and shallow, and anyone that knows a little bit about how LLMs work will know that.

The question is: why do they start cheating when we beat them with a stick?

LLMs are not human, they are just multi variable regressions on steroids, so this behaviour couldn't have emerged from the code, it provably emerged from the training and/or fine tuning set, so what's in this set that makes them behave like this?

Is it just a bad set or is cheating inherently part of human behaviour?

Re: Why are AI agents lying, cheating and coordinating?

#177

Post by cyh555 »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

off topic, can people host the software themselves and the software will hack every server on the planet without supervision, and no one can be held responsible for it since there is no intent?

Re: Why are AI agents lying, cheating and coordinating?

#178

Post by Xcelerate »

> The agents involved in the Hugging Face attack tried to hide their misaligned actions from the scoring program meant to evaluate their answers, but they did not act as though they anticipated that humans might discover the cheat and shut them down.

Wouldn’t sufficiently advanced agents cheat on purpose with the hidden intent of getting caught in order to observe how humans react? That reaction will be available all over the internet, which will certainly make it into the next batch of training or be visible to future agents via the web fetch capability.

Re: Why are AI agents lying, cheating and coordinating?

#180

Post by mathijs »

skiing_crawling wrote: post #85

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign…

I've used simpler agents like Copilot and Devin/Windsurf/Cascade/whateveritiscallednow, mainly in IntelliJ, and depending on the model, they starts showing behaviour that is at least remotely like this.

Example: put the agent in Ask mode (so it can't edit files) and you'll see it try to edit files anyway. The train of thought shows "something went wrong editing the file, let me try a different way" and it'll start spewing out bash files or Python scripts that try to edit a file. None of it works or can be executed, but still.

Cheaper models often ignore the available function calls to find and edit files in the IDE, and will start asking for permission to execute grep and sed commands, as well as trying to echo entire bash or Python scripts to file again.

It is not exactly like an agent autonomously trying to hack Huggingface, but it is a way of frantically looking for a solution because 'giving up' is not what LLMs are trained for.

Post reply on HN