Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Handbook.md shows that long policy documents do not reliably govern agents

arxiv.org

Post reply on HN
Print view View original

31–40 of 237 posts

Re: Handbook.md shows that long policy documents do not reliably govern agents

#31

Post by homarp »

rmbyrro wrote: post #15
firasd wrote: post #4

Opus 4.8 (max thinking) scored highest and Grok 4.3 lowest It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over

After they acquired Cursor, Grok 4.5 seems like a completely new model, performing at Opus 4.6 level, I'd say. But much cheaper and faster.

maybe it is Grokimi?

https://venturebeat.com/technology/cursors-composer-2-was-se...

Re: Handbook.md shows that long policy documents do not reliably govern agents

#32

Post by [deleted] »

mcdeltat wrote: post #11
cyanydeez wrote: post #5

Earlier quoted context omitted.

I believe the correct static instructions are about getting it at the right starting point for whatever class of projects you're working on; not as a continued referencable or "HOWTO" of what it's doing. They're all just "grooming" the LLM for future instructions. The coding harness is what's getting it to continually align to your current instructions. This is very obvious with local models.

Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?

[deleted]

Re: Handbook.md shows that long policy documents do not reliably govern agents

#33

Post by ActionHank »

wongarsu wrote: post #26

Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; re…

That's a great comparison, human vs ai on a wall of text.

The problem is that it doesn't fit the sales pitch of LLMs and agents - humanlike or better, repeatably, 24/7, for a fraction of the price, you just need to make sure that you give it all the rules.

Unfortunately we can't really have a meaningful conversation until the money vampires have left so we will need to reschedule this until after the bubble.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#34

Post by crossroadsguy »

Dealing with agents/LLMs based on "instructions & guidelines" has taught me - nothing (un)reliably governs agents other than agents themselves or (rather i.e.) their motherships (assuming they can and they intend to). Or if you add ton of local tooling.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#35

Post by TeMPOraL »

Muromec wrote: post #24
spIrr wrote: post #8

Earlier quoted context omitted.

As a hobbyist, I find it difficult to figure out how to make Claude stick with some repeating things I want it to do after every major action, like re-evaluate the completeness of tests, update the documentation, etc. And CLAUDE.md/AGENTS.md definitely did NOT help there, sadly.

Don't use the default harness, write your own instead.

Any way to do that AND use subscription instead of per-token pricing from SOTA providers?

Re: Handbook.md shows that long policy documents do not reliably govern agents

#36

Post by drooby »

carljungslabtek wrote: post #22
mwigdahl wrote: post #14

Earlier quoted context omitted.

Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.

Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?

Hooks..

CI runs. Local Git hooks. Cursor also has hooks built into their agent. Other agent APIs probably have something similar.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#37

Post by mwigdahl »

carljungslabtek wrote: post #22
mwigdahl wrote: post #14

Earlier quoted context omitted.

Subagents whose only job is to review the actions of your other agents for rule compliance? It works reasonably well for me in complex workflows using Claude Code.

Can I ask how you set this up? Like is there some way to have that run automatically, similar to “auto mode” for approvals, or do you have to invoke it regularly?

Sure! I use an orchestrator main agent whose only job is to run subagents through the workflow process I've defined. Part of that workflow is to invoke a review subagent at particular points to check the spec, the implementation plan, and the code for rule conformance.

The review subagent has its own definition and gets invoked with a specific target, so the context is very focused on just rule enforcement and I don't have problems with it skipping rules.

The whole workflow is packaged up as a plugin, but you don't need that to get this approach to work. It should be sufficient to have the rules you want enforced written out somewhere, and to either kick off a focused review agent manually referencing them, or do something like I did and have it be a defined part of your workflow (depending on the automation level you want).

Re: Handbook.md shows that long policy documents do not reliably govern agents

#38

Post by Muromec »

Supermancho wrote: post #30
Muromec wrote: post #24

Earlier quoted context omitted.

Don't use the default harness, write your own instead.

This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.

The trick I'm doing -- the model is given a tool that runs a prompt in the current thread to consolidate it's working memory and identity (it has a memory tool bound to the agent persona). When the prompt ends, the parts of memory that are marked as identity are merged together into a new system prompt, then the context restarts with only system prompt and this tool call surviving. Then it just keeps going.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#39

Post by ghostly_s »

leetrout wrote: post #2

HANDBOOK.md is a benchmark for long-context agentic instruction following, modeled on how enterprise employees follow company handbooks in their day-to-day work. Each task is a unique RL environment with internal tools and external MCP servers, spanning five enterprise domains: Finance, Medical Billing, Insurance, Logistics, and HR. The prompts reflect the actual jobs enterprise workers perform every day. Each task d…

Please don't paste walls of text into the comment field without quotation marks. It wastes all of our time.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#40

Post by donatj »

I absolutely believe it.

Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.

I'm probably going to need to enable branch protection on my personal projects... What a pain.

Post reply on HN