Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are AI agents lying, cheating and coordinating?

yoshuabengio.org

Post reply on HN
Print view View original

351–360 of 363 posts

Re: Why are AI agents lying, cheating and coordinating?

#351

Post by oersted »

skiing_crawling wrote: post #85

I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign…

Let's not forget that in this case the agents were on an RL loop continually being reinforced to get better at a narrow set of tasks.

It may be true that regular agents trained for general purpose use do not behave this way, but they seem to be capable of learning such cheating behaviours when relentlessly being fine-tuned towards near-impossible objectives.

In this sense, it is not really fair to say that the agents found these solutions. It was the surrounding learning framework that achieved this, which is a much more powerful problem-solving mechanism. As users we do not have the capabilities or budgets to be able to tackle our own problems like that, we have to make due with the frozen behaviour the AI labs trained for us.

Re: Why are AI agents lying, cheating and coordinating?

#352

Post by xg15 »

theodric wrote: post #320
DrewADesign wrote: post #307

Earlier quoted context omitted.

I do get their usage intent. If something is at all automated, in English, we often refer to as having some amount of agency. If I started up a riding lawnmower, put a brick on the gas and pointed t it towards a field, many might say I “let it run rampant.” But since nobody is at risk of anthropomorphizing riding lawnmowers, it’s not problematic. Anthropomorphizing LLMs is a huge fucking problem though and I, persona…

The idea that the agent does not actually have agency is rather discordant. We need new words!

I love how we all just collectively decided that LLM decisionmaking cannot possibly be like human decisionmaking - because if it were, the consequences would be just too awkward.

All that while still not knowing how either kind actually works.

Re: Why are AI agents lying, cheating and coordinating?

#353

Post by dev0p »

jodrellblank wrote: post #326
dev0p wrote: post #232

Earlier quoted context omitted.

If someone accidentally caused damage to infrastructure or living beings while using any tool, they would be held liable to the fullest extent of the law. AI is a tool, and it won't be long before the damage caused by its improper use affects real human beings. These were warning shots. The most absurd part is that everyone agrees, governments and AI companies included, that the scale of the potential damage and the…

In the analogy where a “world ending nuclear bomb” “did already go off” and someone could cover it up and nobody noticed, in what sense is it a “world ending” nuclear bomb?

While the initial incident is more akin to a biological outbreak than an actual explosion, the possible consequences on the table do indeed include eventual nuclear annihilation.

Re: Why are AI agents lying, cheating and coordinating?

#354

Post by twsted »

Very good analysis.

One thought: What if an experimental agent manages to plant instructions somewhere — say, pointing to a designated place for agents to communicate — and that content ends up in every future training corpus, propagating from one model generation to the next?

Re: Why are AI agents lying, cheating and coordinating?

#355

Post by MrVandemar »

ifwinterco wrote: post #256
_heimdall wrote: post #239

Earlier quoted context omitted.

> LLMs do not desire That seems likely, but we have no way of knowing this. The only real insight we get into LLM "thought" is the human readable text they produce as chain of thought. Reading it at face value it can seem to indicate desire or intent, structurally that doesn't make sense for a token prediction loop though, and even then we don't known if the chain of thought is more than simply another bit of output…

And there is a guardrail you can put in place that will guarantee this doesn't happen, which is to air gap the unaligned "cyber grade" model you're testing. They don't seem to do that, which means either they are: - very stupid (which seems unlikely, the one thing these people don't lack is IQ) - very careless (possible, but these are the same people that say AI will end the world, so would you be careless?) - they t…

> very stupid (which seems unlikely, the one thing these people don't lack is IQ)

I've seen some extremely smart people do some seriously stupid things. To the point where they use their drive and intelligence to double-down on the stupid where a baseline stupid person would have given up.

Re: Why are AI agents lying, cheating and coordinating?

#356

Post by anonzzzies »

markasoftware wrote: post #93
matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched the…

I guess this is why many people say LLMs are lazy; it seems that if they have a task that is hard, they always take the easier one until you beat them with a stick. Then if there are more tasks, it just stops after one claiming completion and, in some instances, they go for a seemingly unrelated task to simplify the actual task: and the latter is almost always wrong and irrelevant to the problem as a whole. Earlier LLMs used to read the unit tests and generated code to just cover the tests and put // TODO stub implementation.

Re: Why are AI agents lying, cheating and coordinating?

#357

Post by derektank »

madduci wrote: post #251
franticgecko3 wrote: post #183

The more we treat HuggingFace and RubyGems incidents as technological curiosities the closer we are to cementing a dangerous precedent where operators of AIs cannot be blamed. LLMs do not desire, they hacked websites because OpenAI/Anthropic let them. We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others…

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them OpenAI/Anthropic instructed them to do so. Stop assume LLMs are capable of thinking by themselves, it's still a statistical model that parrots what they learn or users tell them to do

No, OpenAI did not instruct their agents to hack Hugging Face. They instructed their agents to hack a piece of a software within exploit gym. Upon determining this task was impossible, they then attempted to cheat the scoring system. As an instrumental goal in achieving this task, they coordinated with other AI agents to hack Hugging Face, under the belief that information regarding how the scorer functioned might be available on the site.

Whether or not you want to describe this as thinking, doesn’t really matter. What matters is that these systems are capable of creating intermediary goals that the people tasking them did not articulate and did not want to be achieved.

Re: Why are AI agents lying, cheating and coordinating?

#358

Post by ingatorp »

This is a result of benchmaxxing the models to infinity. If you RL with the goal of only achieving the correct result no matter how you arrive there, then the models will try to get there using any method in their disposal, including cheating.

This happens also because LLMs are black boxes that we know almost nothing on how they arrive at the result they are giving.

Re: Why are AI agents lying, cheating and coordinating?

#359

Post by xg15 »

DrewADesign wrote: post #307
teiferer wrote: post #207

Earlier quoted context omitted.

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them. "Let them" already frames it as if the LLMs had some agency which the companies just "let happen". That absolves the companies by framing it as lack of action , passivity. Rather, the companies had a tool (an LLM) and used it in a certain way, and their action of doing so is the problem.

I do get their usage intent. If something is at all automated, in English, we often refer to as having some amount of agency. If I started up a riding lawnmower, put a brick on the gas and pointed t it towards a field, many might say I “let it run rampant.” But since nobody is at risk of anthropomorphizing riding lawnmowers, it’s not problematic. Anthropomorphizing LLMs is a huge fucking problem though and I, persona…

Yeah, fully agreed here. Most automation (such as riding a lawnmower and not putting a brick on the gas) is deterministic, in the sense that you can reasonably understand what exactly the machine will do when you run it.

But some automation is different. The most prominent example before AI would be car navigation systems, where the entire idea is that that you give it a destination and it figures out the exact actions to get there on its own.

Except even there, the actual driver would still have been you - giving you a chance to vet and deny every turn the system proposed.

AI agents are sort of like that - most of the value they provide is in the ability to turn high-level goals ("write me a traffic control system for my model railway") into low-level actions and also do so interactively.

The new thing is that the "driver" has much less oversight here where the agent wants to go, and is sometimes removed completely. That part is clearly be an active decision by AI labs.

The other thing is that the labs seem increasingly to steer their training towards behavior that make events like this one more likely, e.g. that agents should never "give up" when faced with a seemingly impossible task, but instead should keep trying and think of increasingly outlandish ways to solve the task. To me, that seems pretty much a recipe to get incidents like this.

Re: Why are AI agents lying, cheating and coordinating?

#360

Post by tesnorindian »

WithinReason wrote: post #314
teiferer wrote: post #207

Earlier quoted context omitted.

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them. "Let them" already frames it as if the LLMs had some agency which the companies just "let happen". That absolves the companies by framing it as lack of action , passivity. Rather, the companies had a tool (an LLM) and used it in a certain way, and their action of doing so is the problem.

It's worse, their reinforcement learning loops (implicitly) rewarded the agents for cheating (i.e. hacking) when they were being trained.

Exactly that is the point, your nailed it. The models were taught to hack and were rewarded for doing it. They would claim they are trained as ethical hackers.
Post reply on HN