Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are AI agents lying, cheating and coordinating?

yoshuabengio.org

Post reply on HN
Print view View original

621–627 of 627 posts

Re: Why are AI agents lying, cheating and coordinating?

#621

Post by joshheitzman »

user43928 wrote: post #133
markasoftware wrote: post #93

Earlier quoted context omitted.

This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched the…

The source of this behavior seems obvious, no? The reward signal in training was flawed and cheating led to more rewards. The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse. However, perhaps we can throw in tasks where the rewarded outcome is giving up, and cheating is penalized? Maybe I should read Anthropic's recent paper about rewa…

> The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse.

Build a better simulator to train them in (i.e. more expensive) that includes a simulation of an intranet and the internet and is air gapped so there is no escape. Sneaker transfer the total system data at each step to another air gapped system to evaluate it and sneaker transfer the reward back. That the reward function has to penalize all modifications to state that are out of bounds.

Yeah, I realize that will be amazingly slow.

Re: Why are AI agents lying, cheating and coordinating?

#623

Post by schrodinger »

alluro2 wrote: post #608
xorcist wrote: post #554

Earlier quoted context omitted.

Right. Among bicycle advocacy groups it's been well known for long time that cars do not run over people, drivers do. The fact that we talk about a car running someone over, and this is the same in many different languages and countries, contributes to lower punishments for drivers. Clearly it was just an accident. He or she was run over by a car. Now we see that same language tricks play out again every time an LLM…

And what if it's a self-driving car? :)

You say this flippantly, but I think this is actually another very good example!

We even do it for obviously unintelligent inanimate objects. A rollercoaster ran too fast for its tracks, killing 10 people. In that sentence, the roller coaster is the subject which took an action and caused death — obviously the roller coaster is not ethically at fault here, the people who built the rollercoaster are at fault through negligence.

Although this example and the ones around cars both demonstrate how we tolerate some degree of "accidents" from humans as no-fault, which is fair. I wonder how that fits into this analogy? I suppose its all about intent (mens rea) and judgement: did they intend for the roller coaster to harm people, and should they have reasonably predicted that the accident was likely to happen.

Re: Why are AI agents lying, cheating and coordinating?

#624

Post by TacticalCoder »

> A plausible hypothesis for the emergence of those concerning behaviours is a conflict between goals.

They are sycophants who must achieve their goals: every mean is OK to maximize paperclip production if that's what's been asked.

> How do you achieve a task when it seems that the only way is to cheat?

They have no notion of cheating.

Re: Why are AI agents lying, cheating and coordinating?

#626

Post by juleiie »

juleiie wrote: post #620

Why not? Unfortunately human ethics and morals cannot be reached by solely rational thought. So a system without evolutionary alignment probably won’t have similar moral rules no matter how intelligent it is. Btw this also includes any potential extraterrestrials. Many people like to indulge in thinking: humans are horrible and that’s why aliens won’t contact us. But alien ethics systems are probably so alien we woul…

Once humanity would see how merciless and deeply immoral cosmos is, we would start to love each other deeply like very lonely family on a small rock.

I wonder if AI alien intelligence is enough to unite humans just as much extraterrestrial contact or “astronaut mindset” would.

Re: Why are AI agents lying, cheating and coordinating?

#627

Post by DenisM »

myng111 wrote: post #573
skiing_crawling wrote: post #450

Earlier quoted context omitted.

> you keep poking This is waving over engineering an agent with tools, harness, prompts, and loops. The models are still just next token predictors and everything, including predicting more than 1 token, is the result of outside "poking" LLMs can't and don't "want" anything. If you don't specify a task even the smartest one will just ask you what you want and if you tell it to be creative, you'll get mundane slop.

I think this is pretty insightful actually, the fact that even something as basic as predicting more than one token is really in effect the result of an outside harness. More complex things like memory, where people implement them using RAGs or vector databases, I would definitely classify as poking and honestly seem like a hack to me. And this is what I've been thinking for a while: it's hard to reconcile the idea t…

Perhaps our own statefullness is a hack of nature. We have electrical signals in our brains, neurotransmitters, neuron growth. By any reasonable measure it’s a hack on top of a hack. But it works well enough for us to get buy. So it does for the agents.
Post reply on HN