Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Handbook.md shows that long policy documents do not reliably govern agents

arxiv.org

Post reply on HN
Print view View original

181–190 of 237 posts

Re: Handbook.md shows that long policy documents do not reliably govern agents

#181

Post by pmarreck »

PunchyHamster wrote: post #60
pmarreck wrote: post #28

Earlier quoted context omitted.

You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass. In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(…

So you are essentially saying "you can't, you can only safeguard from effects of LLM eventually ignoring it"

Just like humans, yes.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#183

Post by pessimizer »

john_strinlai wrote: post #165
nzeid wrote: post #158

Earlier quoted context omitted.

This is a bit of a strawman. The parent comment wasn't suggesting that the inherent defects magically disappear nor was it suggesting that local inference today is sufficient for all tasks. At the heart of it, self-hosting liberates your use cases from all the horribly opaque configuration, shadow prompting, etc. And local models are only getting better and more diverse every month.

> The parent comment wasn't suggesting that the inherent defects magically disappear that is almost exactly what they said though..? " Want it to go away, almost like magic? Local inference. [...] all of the common LLM defects will go away.

"[...]"

-----

edit: I suppose I have to spell this out.

"[...]" = "When its under your control, and [you're] no longer being forced to hold it wrong," ≈ "liberates your use cases from all the horribly opaque configuration, shadow prompting, etc."

and

"Almost like magic" is cliché hyperbole. It also means "not magic" in the same way that "almost like a dog" means "not a dog." It was immediately followed by the explicit clarification that was surgically omitted above, in absolute and conscious bad faith.

Hence, "[...]"

Re: Handbook.md shows that long policy documents do not reliably govern agents

#184

Post by ux266478 »

Aurornis wrote: post #173
ux266478 wrote: post #150

Earlier quoted context omitted.

And I think you didn't understand what I wrote, so let me reiterate: Of course policies are equally ineffective at strictly governing the behavior of local models. Why would that change? The actual problem is that somebody is attempting to misuse them for that in the first place. There are only two reasons for it: - They have no other options OR - They have no idea what they're doing Local models solve the first prob…

> In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian". No, I’m talking about the idea up thread that having control over more settings removes all LLM defects with local models. I have all the control I want over the local models they run. No combination of settings i…

He didn't say that, and I didn't say that, because in fact neither of us were talking about just settings, so that's a pretty curious idea to be addressing.

> No combination of settings is going to make them outperform Anthropic’s models.

And even this is addressing something nobody was saying, and manages to come off as a marketing read.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#185

Post by satvikpendem »

ux266478 wrote: post #87
satvikpendem wrote: post #79

Earlier quoted context omitted.

How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.

As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline? Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like tha…

Changing the weights and controls does not alleviate the context limits causing LLMs, local or cloud, to forget over time. I was asking how local LLMs would amelioratile this but looks like there is no solution.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#186

Post by georgesequeira »

victorbjorklund wrote: post #29
spIrr wrote: post #8

Earlier quoted context omitted.

As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.

Hooks can be pretty useful for that. A hook when it is finished ”run tests suite and check coverage” ”check if your changes require updating the docs”

[dead]

Re: Handbook.md shows that long policy documents do not reliably govern agents

#188

Post by jolaflow »

twosdai wrote: post #73

This article to me also implies that there are some potential issues with large Spec based development flows, which I haven't been able to pin down lately. Specifically, having agent implementation drift from the Spec.

I have run into the same problem, and started calling the phenomenon vision drift.

I noticed that the issue tracker I built was well suited to address this problem, as it handles board time-travel. So a simple command like ":replay 4h" gives you a clear overview of how the workflow has evolved over time, and you can checkout previous states at will.

Wrote more about it here:

https://dev.to/ljtn/vision-drift-addressing-the-next-problem...

Re: Handbook.md shows that long policy documents do not reliably govern agents

#189

Post by pmarreck »

dluxem wrote: post #98
Muromec wrote: post #58

Earlier quoted context omitted.

>In this case, it is an advisory control only, because the LLM can also unhook that hook you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.

This appears to be what the claude code-review is doing: https://github.com/anthropics/claude-code/blob/7ef6eec9d9ba8...

this only checks whether the code is in compliance with CLAUDE.md though

Re: Handbook.md shows that long policy documents do not reliably govern agents

#190

Post by pmarreck »

QuercusMax wrote: post #97
pmarreck wrote: post #28

Earlier quoted context omitted.

You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass. In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(…

I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`). That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generall…

Or just make it so that the commit hook can't be edited or bypassed by Claude?
Post reply on HN