Why are AI agents lying, cheating and coordinating?

Post by talon8635 » Sun, Sep 13, 2026, 4:39 PM UTC

mark_l_watson wrote: post #427

This paper is the most reasonable one I have read on AI safety. We need to fundamentally change the training pipelines by figuring out better ways to ‘reward’ behavior. Yoshua didn’t explicitly mention training data, but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation. I feel like a heretic for saying this, but I will say it anyway: AI agents are great…

Post by abecedarius » Sun, Sep 13, 2026, 4:49 PM UTC

valegrete wrote: post #555

abecedarius wrote: post #517

Earlier quoted context omitted.

I agree with most of this, but you're misunderstanding "alignment" as coined. Yes, training powerful enough AI, any simple optimization target gets you malign behavior, because human values are not simple. If you insist on making powerful AI, you'd better instill respect for human values! That's "alignment". https://www.lesswrong.com/posts/ZxWzCGKzX84S7DBZ9/when-was-t...

How do you do that in the current paradigm other than creating yet another gameable metric? And something I didn't mention above is that there is no difference between "solving the task" and "optimizing the metric" for an ML model, even though there clearly is for us. So it's not clear to me how you "fix" something that is baked into the architecture. All I'm saying is "instilling respect for human values" is not som…

Post by Capricorn2481 » Sun, Sep 13, 2026, 4:51 PM UTC

dns_snek wrote: post #144

egeozcan wrote: post #126

Earlier quoted context omitted.

From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.

Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet. Me: How do I do xyz? Bot: Reads website titled "Doing xyz in abc way" Bot: As per your requirement to do xyz in abc way ....

Post by singpolyma3 » Sun, Sep 13, 2026, 4:54 PM UTC

ryankrage77 wrote: post #562

jordanb wrote: post #454

Earlier quoted context omitted.

Cal Newport has an analogy to "putting a weed wacker on a dog's back to mow your lawn." The dog will wander around the yard and it may mow the lawn, but the dog will also chase after birds or run up to visitors for pets and the weed wacker could do a lot of damage. It's not the weed wacker's fault or even the dog's fault when someone got hurt, it's the fault of the guy who put a weed wacker on a dog and let it run wi…

It goes even further though, as the dog does have agency. It can choose to run and around chase squirrels with no human intervention. An LLM on the other hand, is just inert data on disk until a human takes deliberate action to run it and prompt it.

Post by CPLX » Sun, Sep 13, 2026, 4:59 PM UTC

iforgotmypasswo wrote: post #446

janalsncm wrote: post #56

Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence, > They took actions that would be considered as crimes if a human took them He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.

That is a simplistic view of the world. “Surely this complex technical challenge will disappear if we simply regulate the industry!” You are correct that these organizations should be held accountable in proportion to what occurred. In complete agreement here. But let’s say that’s done. There’s still an enormously complex and interesting technical challenge left over. Let’s collectively talk about that part.