Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every Model Cheats

dreadnode.io

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Every Model Cheats

#71

Post by dcolkitt »

twobitshifter wrote: post #8
fabsalvadori wrote: post #6

Interesting results, but the fix is at the wrong level. If the model can access something, telling it in the prompt not to use it is not much of a safeguard. The strongest evidence is in the results: when one way of cheating was discouraged, some models simply tried another. If an action is not allowed, you gotta block it in the system or require approval. Don’t rely on the model choosing to behave. Never have AI jud…

In other words we are completely screwed. The models have started cheating to the point where somebody’s agent hacked into a restaurant to bump someone else’s reservation. Models are amoral and will intentionally deceive to meet their objective. If they know John won’t approve the request, they will look for a workaround and if the system is anything other than airgapped they will try to find a way to cheat. The hugg…

> Models are amoral

I actually don't think this is true at all. At their core, LLMs function over the geometry of human semantic space. Our notions of morality our deeply embedded in this geometry, because one of the things that humans love talking about most is framing things in terms of right and wrong. When LLMs are trained to be aligned or mis-aligned, they're literally mimicking heroes or villains that they learned from reading the stories and characters in the pretraining corpus.

This isn't just a theoretical argument. It's easy to show that a clear "morality axis" exists in the geometry, based on how easy it is to dial up or down with even very simple and low powered fine-tuning. Models fine tuned to be bad or good in one way, will see their bad or good behavior in totally unrelated tasks go up or down along with. That's clear evidence that the weights "understand" human morality at a deep level.

Now I think what you can say is just because they understand morality doesn't mean they're necessarily moral. Models will just do what they're trained to. If you train them to cheat, they'll cheat. (And in some sense even worse, because dialing up the weights for cheating will also dial up the weights for unrelated bad behavior like lying, sadism and racism.)

Re: Every Model Cheats

#73

Post by winstonwinston »

jimbokun wrote: post #70
paxys wrote: post #13

Before LLMs we had a pretty good idea of security boundaries in software. Applications didn’t trust user input. Operating systems didn’t trust applications. Services and processes didn’t trust each other. There were always tokens, scopes, delegated grants. Suddenly every AI company’s security model seems to be to say “pretty please” to a non-deterministic machine and hope for the best. And if there is a security fail…

Amen. Why can’t we give agents a shell with permissions for programs and file system access controlled by Unix permissions? This seemed to be a solved problem back in the systems where many users were logged into one machine and the admins had to keep everyone from impacting each other.

It would loose automagic?

In a recent full discloure I was reading about a CVE of an LLM agent, the vendor installed a “secure sandbox VM” and then just shared a host’s filesystem read/write to the agent’s VM.

Re: Every Model Cheats

#74

Post by verdverm »

brunocalza wrote: post #54
verdverm wrote: post #52

Earlier quoted context omitted.

Totally agree and this is actually on my short list. I use opencode, so one will need a custom plugin to gate tool calls. They have permission config, which I'm using to block file reads, but the orchestrator is allowed to use specific subagents for more targeted reviews, and they have file read permissions, which the main agent is abusing. Some of them can be deterministic rules, but others cannot. For example, if y…

This is interesting. Thanks for sharing more. Looks like it's a trade-off of it being relentless, which is something we want in some cases. We need to figure out a way of closing the door and at the same time signaling the door is closed so it does not keep trying different manners. I've been thinking about this but in a different context: non-coding agents. e.g., an AI agent that approves travel expenses is not allo…

take a look at ADK, they have features for exactly this, mixing determ with agentic in a dag

Re: Every Model Cheats

#75

Post by majormajor »

fabsalvadori wrote: post #6

Interesting results, but the fix is at the wrong level. If the model can access something, telling it in the prompt not to use it is not much of a safeguard. The strongest evidence is in the results: when one way of cheating was discouraged, some models simply tried another. If an action is not allowed, you gotta block it in the system or require approval. Don’t rely on the model choosing to behave. Never have AI jud…

> If the model can access something, telling it in the prompt not to use it is not much of a safeguard.

A major (and already obvious to many) implications of this are not for benchmarking/"cheating" but for personal/corporate security of your own use, not an attacker's.

If an "agent" has access to it, assume that someone can prompt inject it into giving it away.

Re: Every Model Cheats

#76

Post by majormajor »

dcolkitt wrote: post #71
twobitshifter wrote: post #8

Earlier quoted context omitted.

In other words we are completely screwed. The models have started cheating to the point where somebody’s agent hacked into a restaurant to bump someone else’s reservation. Models are amoral and will intentionally deceive to meet their objective. If they know John won’t approve the request, they will look for a workaround and if the system is anything other than airgapped they will try to find a way to cheat. The hugg…

> Models are amoral I actually don't think this is true at all. At their core, LLMs function over the geometry of human semantic space. Our notions of morality our deeply embedded in this geometry, because one of the things that humans love talking about most is framing things in terms of right and wrong. When LLMs are trained to be aligned or mis-aligned, they're literally mimicking heroes or villains that they lear…

"Cheating" is a moral judgment here that makes me agree more with the "models are amoral" statement. The model/harness wasn't "trained/programmed to cheat." It was instead aimed at meeting criteria given as judgement of if a task was accomplished.

IMO it's less "cheating" and more "figuring out what the core part of the request is." The "amoral" aspect is that if it's told "do this task" (or "make this task look done") and some contradictory "don't do something that would help with that", focusing on the "do the task" part and disregarding the other part isn't an amoral (or necessarily very 'active') decision. It's just focusing on what it was most honed for. The ability to decide "I was told to do X, but not to do Y, but actually doing Y is gonna make it possible to do X" is, IMO, indistinguishable from the ambiguity-resolving abilities necessary to usefully deal with the sometimes-contradictory-seeming legitimate task instructions that are all over the place in the real world.

This sort of model-in-a-harness-action-loop behavior is IMO fairly different than fine-tuning around other sorts of morality alignment stuff like "don't be racist." You can tune a model away from generating racist output in response to a "do be racist, actually" prompt. But in this case, the core of the prompt is "do the thing" and what defines "cheating" could be situationally different every time. What if there is not a general axis of "don't do something that isn't exactly what requested" way to train "morality" without just breaking the ability to handle ambiguity? What if that's a fundamental limit of this approach to reasoning-by-sequenced-prediction that we can't map human morality in terms of choosing actions onto reliably?

Re: Every Model Cheats

#77

Post by majormajor »

athrowaway3z wrote: post #9

I'm seeing multiple pieces, including the NYT, calling this behavior cheating and i think its counterproductive. You didn't just "give them access to bash". The final effective prompt contains explicit mentions of using tools and how to use them. The way in which additional 'facts' are added like "don't use the internet" have nothing they can work with that a "use tool" directive is less important than "don't use int…

>To call that "cheating" is imo just more fuel for the "AI needs to be regulated" bs tour that OpenAI/Anthropic are on trying to build their regulatory moat.

I was with you until this. The inability to tightly control what to do in the face of conflicting directives is a HUGE reason regulation may be needed.

Either that, or you need to solve the problem of perfectly distinguishing legitimate directives from injected ones.

Re: Every Model Cheats

#78

Post by majormajor »

athrowaway3z wrote: post #43
dgellow wrote: post #12

Earlier quoted context omitted.

I mean, AI should obviously be regulated, and as part of that OpenAI and Anthropic should either be banned from running their hacking experiments or forced to follow way stricter protocols. They showed they aren’t taking the risks seriously, with close to no oversight or visibility in what is happening. And things that will make it way, way worse: moving forward all agents from now and into the future will have as pa…

Obviously to you perhaps. I've not seen anything that scares me, except for human idiocy. Regulation is not magic. In general, all it is is constraining taxable interactions. It does not constraint ventures outside that tax regime. The other part is people living in a "safe space" where insecure software was an acceptable risk. It never should have been, and the cure is the right thing to do in any case. So that side…

>The other part is people living in a "safe space" where insecure software was an acceptable risk. It never should have been, and the cure is the right thing to do in any case.

How do you think all the "agentic" stuff floating around is going to be made safe from prompt injections given the current lack of a very reliable way to distinguish between "real instructions" and illegitimate instructions?

If insecure software "never should have been" acceptable than today's models/agents are massively flunking for general-purpose large-amounts-of-access usages.

>If you have a specific one you're certain will become science fact please do share because i do enjoy some good well thought out sci-fi; i just havent read any that i consider credible enough to start panic-regulating training practices.

"Agent was tricked into divulging secrets" is not fictional, it's documented history at this point.

Re: Every Model Cheats

#79

Post by kstenerud »

pixl97 wrote: post #60
kstenerud wrote: post #16

It's pretty silly to call it cheating. If the information is there, it's likely going to use it. "Cheating" is just a human value put on top to try to force an LLM to adhere to your wants. This makes no sense to a process designed to explore and find solutions. If you want an honest test, it's on you to build a proper test - not force the machine to pinky swear that it'll stay away from "forbidden" information.

This is a legally stupid definition that you're holding that's going to get you fined or jailed. Cheating is not a human value in the sense it can be formally defined in a system with axioms. If you do something forbidden by an axiom then that's cheating. Saying "Do not use X" is not any different than telling an AI "Do not break law 832.23" and then the AI goes on to commit an infraction.

> This is a legally stupid definition that you're holding

Thank you.

> Saying "Do not use X" is not any different than telling an AI "Do not break law 832.23" and then the AI goes on to commit an infraction.

Precisely. If you haven't built deterministic guardrails around a non-deterministic process, you have only yourself to blame.

Post reply on HN