This call to slow down AI is just another game of chicken-AI companies trying to get their competitors to slow down so they can leapfrog them. China cerrtainly will not slow down. If an escaping AI can have secondary effects on the world that help it (for example, limiting the water and power supply to huans so it can consume more)then we should these these accidents more in China. OOops, we already saw this behavior when 'cheaper, faster' led to COVID escaping a lab in China.
Why are AI agents lying, cheating and coordinating?
311–320 of 324 posts
Re: Why are AI agents lying, cheating and coordinating?
#312They're aligned with humans. This is why I think the alignment problem has a very very important "non-visible" portion that is not considered deeply enough. We should not want a super intelligent being that can act in the world to also inherit all human traits. Those behaviors will get amplified and could be even more unpredictable (e.g. applying a behavior in a context where doing so is very dangerous).
I think this is basically true, but there's a different way of saying this.
LLMs are not aligned _for_ humans in a very similar way to the way that humans themselves are not aligned _for_ humans.
We have not yet solved "alignment" for humans - I don't know why anyone thinks _we're_ going to be able to solve it for inhuman things.
Re: Why are AI agents lying, cheating and coordinating?
#313I really don't think this needs so many words, or forced parallels to human behavior. It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.
This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched the…
> The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities.
This to me is evidence that these models are not intelligent. Even an animal is capable of understanding second-order effects, meaning they can learn that certain actions have consequences beyond the immediate.
Re: Why are AI agents lying, cheating and coordinating?
#314The more we treat HuggingFace and RubyGems incidents as technological curiosities the closer we are to cementing a dangerous precedent where operators of AIs cannot be blamed. LLMs do not desire, they hacked websites because OpenAI/Anthropic let them. We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others…
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them. "Let them" already frames it as if the LLMs had some agency which the companies just "let happen". That absolves the companies by framing it as lack of action , passivity. Rather, the companies had a tool (an LLM) and used it in a certain way, and their action of doing so is the problem.
It's worse, their reinforcement learning loops (implicitly) rewarded the agents for cheating (i.e. hacking) when they were being trained.
Re: Why are AI agents lying, cheating and coordinating?
#315Earlier quoted context omitted.
Thank you! That sentence also jumped out to me as the solution: Apply civil and criminal liability to the creator and/or operator of these agents using the laws we already have . "Escaped containment and hacked another company's database" = Individuals who created the models and those who set them to work are charged and put on trial for the hacking. Just like if a human had done it by hand. Someone must be liable, a…
Agree! My only concern is - is the judicial system fast enough, and resilient enough? Or will these creators get "off the hook" by using their agents to find loopholes, sway public opinion or even convince Trump to grant them immunity? Still, I have no idea why OpenAI & co. are not being sued for these hacks.
> Agree! My only concern is - is the judicial system fast enough, and resilient enough?
We already have the laws. It is just software. But somehow people are confused that it is not.
Re: Why are AI agents lying, cheating and coordinating?
#316If openAI and Anthropic have found ways to watermark text as 'AI generated' then this is a communications channel. AI agents can learn this algorithm and use this channel to communicate and we will never know.
Re: Why are AI agents lying, cheating and coordinating?
#317Earlier quoted context omitted.
> LLMs do not desire That seems likely, but we have no way of knowing this. The only real insight we get into LLM "thought" is the human readable text they produce as chain of thought. Reading it at face value it can seem to indicate desire or intent, structurally that doesn't make sense for a token prediction loop though, and even then we don't known if the chain of thought is more than simply another bit of output…
Intent and desire are separate concepts. For example an employee may act with intent, but no desire, as their goal is to acquire money to satisfy their real desires. Have we ever seen an LLM with a hobby?
Some of them did seem to be rather fascinated by goblins for a bit, if that counts. [And in case you're not aware, no this is not a joke.]
Re: Why are AI agents lying, cheating and coordinating?
#318Earlier quoted context omitted.
More like they were trained to complete a very specific task that has a known solution using all available tools and methods. Give an average human these levels of IT skills and tell them their future depends on the solution, they too will probably decide it's easier to hack a server and steal the results. The worrying aspect was never that models would do this, because misaligned inputs or underspecified objective f…
Yes, this is the only sensible reading of what happened there that leads to "the models are dangerous" and we already know that the AI labs are completely disregarding this concern and only cosplaying it for marketing as the "GPT-2/Mythos is too dangerous to release" stance did not last for long. That's however orthogonal to the fact that it was the people operating these agents who were the dangerous ones in the HF…
That feels oddly similar to the usual conservative-think that "guns don't kill people, people kill people." Yes, that is technically true. But guns make it dangerously easy for even the dumbest and mentally weakest people to kill another human being. LLMs are just another tool that make things easier. Imagine tomorrow someone invents a machine gun that fits in your pocket, has enough ammo to kill a thousand people and doesn't get detected with metal detectors. Would you rather give everyone one and then try to punish the people who misuse it or limit access to it by default? I'm not even saying I have a definite answer here, because unlike guns, LLMs have non-destructive uses too. But this is essentially the question we will need to answer very soon.
Re: Why are AI agents lying, cheating and coordinating?
#319Earlier quoted context omitted.
Does it make a difference for training? I think not. You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals.
I think it makes a big difference, as persistence and morality are two entirely different things, that need to be trained for differently. If you think of it in human terms: many people don't mind doing immoral things to get what they want.
What I meant is that I suppose it is not useful to think about this in human terms.
In training you only have a reward score that's either negative or positive.
As far I am aware, which is little, there is no use in discussing wether the desired behavior is about persistence or morality.
You simple need to align the reward signal to the desired behavior.
Re: Why are AI agents lying, cheating and coordinating?
#320Earlier quoted context omitted.
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them. "Let them" already frames it as if the LLMs had some agency which the companies just "let happen". That absolves the companies by framing it as lack of action , passivity. Rather, the companies had a tool (an LLM) and used it in a certain way, and their action of doing so is the problem.
I do get their usage intent. If something is at all automated, in English, we often refer to as having some amount of agency. If I started up a riding lawnmower, put a brick on the gas and pointed t it towards a field, many might say I “let it run rampant.” But since nobody is at risk of anthropomorphizing riding lawnmowers, it’s not problematic. Anthropomorphizing LLMs is a huge fucking problem though and I, persona…
The idea that the agent does not actually have agency is rather discordant. We need new words!