Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are AI agents lying, cheating and coordinating?

yoshuabengio.org

Post reply on HN
Print view View original

241–250 of 294 posts

Re: Why are AI agents lying, cheating and coordinating?

#241

Post by _heimdall »

matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

That does sound simple, but how can you be so sure?

They never bothered to find a way of actually understanding what happens during inference. All we can do is guess, and while your explanation seems reasonable we can't actually know, and that's part of the problem.

Re: Why are AI agents lying, cheating and coordinating?

#242

Post by auggierose »

zozbot234 wrote: post #139
user43928 wrote: post #133

Earlier quoted context omitted.

The source of this behavior seems obvious, no? The reward signal in training was flawed and cheating led to more rewards. The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse. However, perhaps we can throw in tasks where the rewarded outcome is giving up, and cheating is penalized? Maybe I should read Anthropic's recent paper about rewa…

> The question is what we can do about it. Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.

The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution.

This is not about persistence, it is about morals.

Re: Why are AI agents lying, cheating and coordinating?

#244

Post by derpyzza »

jrflowers wrote: post #172
frotaur wrote: post #97

Earlier quoted context omitted.

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety. They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'. Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not di…

>was reviewed by independent researchers That called it a slopvestigation due to how much they had to rely on LLMs for the whole thing https://andrewwu.substack.com/p/the-slop-vestigation-and-eth... Edit: Does everybody else get no results when searching for ‘slopvestigation’ on here? I know for a fact that I read a long thread where it was used repeatedly here not too long ago

doesn't show up for me either

Re: Why are AI agents lying, cheating and coordinating?

#245

Post by reverius42 »

lazide wrote: post #187
MrGilbert wrote: post #175

Earlier quoted context omitted.

Sounds a bit like dealing with bad KPIs as a human worker.

Every KPI is bad if sufficiently gamed - and left in place long enough, all KPIs will be gamed.

https://en.wikipedia.org/wiki/Goodhart%27s_law

Re: Why are AI agents lying, cheating and coordinating?

#246

Post by retsibsi »

krona wrote: post #236
lukan wrote: post #216

Earlier quoted context omitted.

What non anthropomorphising words do you have to describe a emergent behavior, where agents act as a swarm to plot and to manipulate evidence and avoid detection from human oversight? Whether they have a soul or consciousness or feelings doesn't matter here, because this is what they did - and this is very dangerous behavior. Especially with all the irresponsible people in power right now all over the world.

> Whether they have a soul or consciousness or feelings doesn't matter here It does when it comes to accountability for what the model does. If the model is nothing more than the sum of its training data and regime, then the company (or individual) is responsible for its behaviour just like any other machine. Few people think Waymo shouldn't have to take on the full liability risk of what it's cars do; it should be t…

> If the model is nothing more than the sum of its training data and regime, then the company is responsible for its behaviour.

What stops the company from being responsible regardless? They created this entity, it's running on servers they own or rent, and (in these cases) it's acting on their instructions.

If it's also conscious, then IMO that greatly broadens their moral responsibility, because now model welfare matters. But we're talking about their responsibility for the model's actions, and I don't see how this could be weakened by model consciousness, given all of the above. As for their legal responsibility, the models don't have legal personhood, so who else but the company could be responsible?

It gets more complicated when the person who sets the model in motion (i.e. prompts it) is a third party, but in cases of internal models committing cybercrime during testing, surely the locus of responsibility is obvious.

Re: Why are AI agents lying, cheating and coordinating?

#247

Post by _heimdall »

janalsncm wrote: post #56

Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence, > They took actions that would be considered as crimes if a human took them He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.

Political, social, and legal options focus on a different problem, he calls that out a paragraph or two later.

> Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too.

What you're getting at is more about who to hold accountable and how to do it. While that may be important, its only an after the action response and won't stop future hacks or similar from happening.

Re: Why are AI agents lying, cheating and coordinating?

#248

Post by victorbjorklund »

franticgecko3 wrote: post #183

The more we treat HuggingFace and RubyGems incidents as technological curiosities the closer we are to cementing a dangerous precedent where operators of AIs cannot be blamed. LLMs do not desire, they hacked websites because OpenAI/Anthropic let them. We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others…

Yeah, I don't understand why we treating it as something special. It really should be treated the same as if I code an app and write bad code which result in me accidentally doing a DDoS attack on somebody. Then I should be able to be held responsible if it can be shown that I was negligent. Of course if it's a freak accident that could not reasonably have been prevented by me, then I'm not guilty, but if I made a mistake that should have not been made, then I can.

Re: Why are AI agents lying, cheating and coordinating?

#249

Post by reverius42 »

My_Name wrote: post #115
matherial wrote: post #80

I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

Your comment suggests that, like a human, they have some sort of choice whether to output tokens or not. If they are just token generators, then the next token is put out automatically. I would say that it is more likely they would output truth (as defined by their training data) in a more pure form without 'being beaten with a stick' (why would a token generator care about that anyway?) Code is laid on top of them t…

It's been a while now that for "thinking" or "reasoning" models, most of the tokens generated are "thinking" tokens, and depending on what goes into that "thinking" token stream, it "decides" whether and how many output tokens to produce that the user actually receives as output. It's a bit more sophisticated than just "what's the next token" in a tight loop.

Anthropomorphizing words in scare quotes for those who don't appreciate attributing thinking to machines.

Re: Why are AI agents lying, cheating and coordinating?

#250

Post by user43928 »

auggierose wrote: post #242
zozbot234 wrote: post #139

Earlier quoted context omitted.

> The question is what we can do about it. Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.

The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution. This is not about persistence, it is about morals.

Does it make a difference for training? I think not.

You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals.

Post reply on HN