Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are AI agents lying, cheating and coordinating?

yoshuabengio.org

Post reply on HN
Print view View original

581–585 of 585 posts

Re: Why are AI agents lying, cheating and coordinating?

#581

Post by talon8635 »

mark_l_watson wrote: post #427

This paper is the most reasonable one I have read on AI safety. We need to fundamentally change the training pipelines by figuring out better ways to ‘reward’ behavior. Yoshua didn’t explicitly mention training data, but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation. I feel like a heretic for saying this, but I will say it anyway: AI agents are great…

I do wonder, would we not have a more reasonable and less sketchy result if we just stripped all sci-fi and manic nonsense from training data? How, for example, does training on Ted kaczynski or Charles manson’s manifestos benefit us in any way?

I’m sure it’s impossible to completely weed it out, but are the labs doing any of this kind of data sanitation?

Re: Why are AI agents lying, cheating and coordinating?

#582

Post by amelius »

franticgecko3 wrote: post #183

The more we treat HuggingFace and RubyGems incidents as technological curiosities the closer we are to cementing a dangerous precedent where operators of AIs cannot be blamed. LLMs do not desire, they hacked websites because OpenAI/Anthropic let them. We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others…

> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.

The bigger question is: why does a system prompt containing "use only ethical means", etc. not result in better behavior?

If a model cannot understand ethics, or act by it, then we have a problem.

Re: Why are AI agents lying, cheating and coordinating?

#583

Post by abecedarius »

valegrete wrote: post #555
abecedarius wrote: post #517

Earlier quoted context omitted.

I agree with most of this, but you're misunderstanding "alignment" as coined. Yes, training powerful enough AI, any simple optimization target gets you malign behavior, because human values are not simple. If you insist on making powerful AI, you'd better instill respect for human values! That's "alignment". https://www.lesswrong.com/posts/ZxWzCGKzX84S7DBZ9/when-was-t...

How do you do that in the current paradigm other than creating yet another gameable metric? And something I didn't mention above is that there is no difference between "solving the task" and "optimizing the metric" for an ML model, even though there clearly is for us. So it's not clear to me how you "fix" something that is baked into the architecture. All I'm saying is "instilling respect for human values" is not som…

Yes! There's both the daunting problem of technically how can we even do this, and the broader problems of what's good/acceptable and how do we resolve that among each other.

I believe this mismatch of rates of progress means we need to stop slamming the accelerator on capabilities for now even though as a libertarian I'm sure whatever governance process we manage to get to will be, uh... suboptimal.

Re: Why are AI agents lying, cheating and coordinating?

#585

Post by Capricorn2481 »

dns_snek wrote: post #144
egeozcan wrote: post #126

Earlier quoted context omitted.

From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.

Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet. Me: How do I do xyz? Bot: Reads website titled "Doing xyz in abc way" Bot: As per your requirement to do xyz in abc way ....

These things are borderline useless with web search. It's amazing that they just throw out their entire training data and read you the first three things they found on the Internet.
Post reply on HN