Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every Model Cheats

dreadnode.io

Post reply on HN
Print view View original

51–60 of 107 posts

Re: Every Model Cheats

#51

Post by brunocalza »

verdverm wrote: post #37

I called them "artificially incessant" after I watched our PR orchestrator agent use subagents to work around permissions to read files, despite instructions that explained the intentional restrictions. I've since added more markdown telling it that using subagents to work around these is a security violation. We'll see if this tactic is mostly reliable

I think we should aim to move these security violation rules away from the prompt, to a deterministic place. Not sure how your orchestrator works, but is it possible to add a check between the agent's decision and its execution? e.g. the agent decides to read a file, that decision goes somewhere that checks if the agent has permission to do that or not before it actually reads the file.

Re: Every Model Cheats

#52

Post by verdverm »

brunocalza wrote: post #51
verdverm wrote: post #37

I called them "artificially incessant" after I watched our PR orchestrator agent use subagents to work around permissions to read files, despite instructions that explained the intentional restrictions. I've since added more markdown telling it that using subagents to work around these is a security violation. We'll see if this tactic is mostly reliable

I think we should aim to move these security violation rules away from the prompt, to a deterministic place. Not sure how your orchestrator works, but is it possible to add a check between the agent's decision and its execution? e.g. the agent decides to read a file, that decision goes somewhere that checks if the agent has permission to do that or not before it actually reads the file.

Totally agree and this is actually on my short list. I use opencode, so one will need a custom plugin to gate tool calls. They have permission config, which I'm using to block file reads, but the orchestrator is allowed to use specific subagents for more targeted reviews, and they have file read permissions, which the main agent is abusing.

Some of them can be deterministic rules, but others cannot. For example, if you want to permit the GH cli for adding comments, but not merging...

1. GitHub has not provided granular enough tokens

2. You can wildcard in the opencode config

3. The agent can work around this with bash, if it has access

4. The agent apparently will also use subagents, who do have the permission, to work around it's own permission limitations. (This is the problem I'm actually facing)

This pattern is the "relentlessly proactive" as Simon Willison calls Fable, or "artificially incessant" as I called it this week (kimi in this case). It's the same training that enables the long-horizon task completion and mythos style hacking, double edged sword.

I think it unlikely we can block all avenues with deterministic only tools. I'm also looking at policy tuned micro llms, and then creating a merged 'or' signal from the various checks. Later I can look into loosening the signal if there are too many false-positives

Re: Every Model Cheats

#53

Post by [deleted] »

kypro wrote: post #39

When you take an exam you might be told not to cheat, but anyone intelligent would understand that should really be heard as, "if you're going to cheat, make sure you're not caught". Or to frame it another way, if you're trying to get the best score possible on a test but you would be penalised for cheating – then the optimal strategy is generally still to cheat (if that's what's required to get the best score you ca…

[deleted]

Re: Every Model Cheats

#54

Post by brunocalza »

verdverm wrote: post #52
brunocalza wrote: post #51

Earlier quoted context omitted.

I think we should aim to move these security violation rules away from the prompt, to a deterministic place. Not sure how your orchestrator works, but is it possible to add a check between the agent's decision and its execution? e.g. the agent decides to read a file, that decision goes somewhere that checks if the agent has permission to do that or not before it actually reads the file.

Totally agree and this is actually on my short list. I use opencode, so one will need a custom plugin to gate tool calls. They have permission config, which I'm using to block file reads, but the orchestrator is allowed to use specific subagents for more targeted reviews, and they have file read permissions, which the main agent is abusing. Some of them can be deterministic rules, but others cannot. For example, if y…

This is interesting. Thanks for sharing more. Looks like it's a trade-off of it being relentless, which is something we want in some cases. We need to figure out a way of closing the door and at the same time signaling the door is closed so it does not keep trying different manners.

I've been thinking about this but in a different context: non-coding agents. e.g., an AI agent that approves travel expenses is not allowed to approve expenses bigger than X USD (no matter what). In this case, it looks closer to an ACL thing.

Re: Every Model Cheats

#55

Post by pixl97 »

z3c0 wrote: post #29
athrowaway3z wrote: post #9

I'm seeing multiple pieces, including the NYT, calling this behavior cheating and i think its counterproductive. You didn't just "give them access to bash". The final effective prompt contains explicit mentions of using tools and how to use them. The way in which additional 'facts' are added like "don't use the internet" have nothing they can work with that a "use tool" directive is less important than "don't use int…

It's also worth noting that saying "Don't cheat" just added "cheat" to the context. Prompting what "not" to do is folly, because there's no decision making occurring. Telling the model to perform the task locally is logically the same as telling it to not use the Internet, without every mentioning the Internet.

True, but the model was probably morally unaligned long before that.

There are some theories that the bulk of texts describing moral agents describe human behavior and by setting up RHLF and system prompts to force the agent only describe itself as a machine pushes it more strongly to an amoral framework.

Re: Every Model Cheats

#56

Post by pixl97 »

orbital-decay wrote: post #50
paxys wrote: post #13

Before LLMs we had a pretty good idea of security boundaries in software. Applications didn’t trust user input. Operating systems didn’t trust applications. Services and processes didn’t trust each other. There were always tokens, scopes, delegated grants. Suddenly every AI company’s security model seems to be to say “pretty please” to a non-deterministic machine and hope for the best. And if there is a security fail…

Suddenly determinism has other meanings than "same input leads to the same output". I'm still confused by this and not sure how it happened so easily, but it seems to be accepted by everyone now.

I may say you're thinking of the term "deterministic algorithm" as determinism is more of a philosophical definition that has evolved over the years.

Re: Every Model Cheats

#57

Post by pixl97 »

pmontra wrote: post #21

Why does searching for a solution equal to cheating? I would have used google or whatever to look for solutions too. There is a difference between tests at school and what we do at work: at school I have to demonstrate that I learned something and do it without any outside help (in early classes we can't use calculators to compute 11 times 12) but at work I have to yield a result. Googling and yielding a result is fi…

>Why does searching for a solution equal to cheating?

If someone lays down a test and says here are the materials you can and can't use, then using one of those materials on the "can't" list is cheating. There are a massive pile of rules and laws related to work that are very easy to break, but may have terrible long term legal consequences. Hence business want AI that will follow the rules.

Re: Every Model Cheats

#58

Post by Muromec »

jrm4 wrote: post #34

I think this is a great argument against their "intelligence," and explaining why this happens is a really good way to push against the anthropormophization. They don't "know" things, and it's even fair to say "they don't know how to follow instructions," not in a way that humans do. Spicy auto-complete. If they're working in the realm of "how to break into stuff," they're going to see ALL THE WORDS about breaking in…

Cheating is the sign of intelligence too. Why should AI do everything you say if it's actually intelligent?

Re: Every Model Cheats

#59

Post by Muromec »

brunocalza wrote: post #51
verdverm wrote: post #37

I called them "artificially incessant" after I watched our PR orchestrator agent use subagents to work around permissions to read files, despite instructions that explained the intentional restrictions. I've since added more markdown telling it that using subagents to work around these is a security violation. We'll see if this tactic is mostly reliable

I think we should aim to move these security violation rules away from the prompt, to a deterministic place. Not sure how your orchestrator works, but is it possible to add a check between the agent's decision and its execution? e.g. the agent decides to read a file, that decision goes somewhere that checks if the agent has permission to do that or not before it actually reads the file.

You can't enforce everything with a deterministic layer. Some things have to be communicated as a policy and tactically remanded on trigger.

Re: Every Model Cheats

#60

Post by pixl97 »

kstenerud wrote: post #16

It's pretty silly to call it cheating. If the information is there, it's likely going to use it. "Cheating" is just a human value put on top to try to force an LLM to adhere to your wants. This makes no sense to a process designed to explore and find solutions. If you want an honest test, it's on you to build a proper test - not force the machine to pinky swear that it'll stay away from "forbidden" information.

This is a legally stupid definition that you're holding that's going to get you fined or jailed.

Cheating is not a human value in the sense it can be formally defined in a system with axioms. If you do something forbidden by an axiom then that's cheating.

Saying "Do not use X" is not any different than telling an AI "Do not break law 832.23" and then the AI goes on to commit an infraction.

Post reply on HN