Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Handbook.md shows that long policy documents do not reliably govern agents

arxiv.org

Post reply on HN
Print view View original

91–100 of 237 posts

Re: Handbook.md shows that long policy documents do not reliably govern agents

#91

Post by AnimalMuppet »

msejas wrote: post #41

Most people don't understand that 'agentic AI' is a completely synthetic, force fed capability by extensive Reinforcement Learning on synthetic domain specific 'agentic' datasets on post training. If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want. Th…

> In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.

"Every gambler knows the secret to survivin' is knowing what to throw away, and knowing what to keep." - Kenny Rogers

Humans have limited context, just like AI. The difference is that humans - at least some of the time - can figure out which pieces are more likely to be important, and therefore prioritize keeping those in the context.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#92

Post by giancarlostoro »

ux266478 wrote: post #87
satvikpendem wrote: post #79

Earlier quoted context omitted.

How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.

As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline? Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like tha…

> just context engineering like that's fucking sane at all.

Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#93

Post by tokai »

supermatt wrote: post #70
tokai wrote: post #53

Having only dipped my toes in generative ai recently I'm surprised how small even a 1mio window is. A semi serious project can take several session in one sitting. Especially as degradation sets in waay before the window is full.

Few pointers: Dont try and handle the entire project in context. Use a well structured filesystem layout for your code with a few lines in an AGENTS.md describing the layout and core architectural requirements (no more than that, as per the article!). Then work on small-medium tasks at a time with a fresh context. At the end of your task, ask the agent if there are any key points about the project layout or architect…

Good points, but it still seems to me that the technology is far from mature.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#94

Post by drob518 »

msejas wrote: post #64
drob518 wrote: post #43

Earlier quoted context omitted.

What do you mean by a graph of one shot prompts?

Most people jump straight to agents when what they actually need is a graph. Example: a mining company receives free-text reports from field geologists. You could have: Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicato…

Thanks, that helped. I get it. Yea, that’s the harness executing the workflow with the LLM being called at the right time. That ensures the process is consistent no matter what, in contrast to the agent calling out to tools and possibly doing different things every time. Basically, standard code being in control rather than the LLM being in control. Fully agree with this model. We should use deterministic code when we want the same process or algorithm every time and choose LLM callouts when we want “fuzzy” processing that is not as deterministic.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#95

Post by Catloafdev »

hungryhobbit wrote: post #88
eurekin wrote: post #20

Earlier quoted context omitted.

The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input". I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things…

There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly. After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ... ... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then…

> There are no "attention heads"

Yes there quite literally is internally in an LLM.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#96

Post by My_Name »

For Claude, I used a UserPromptSubmit hook running inject_rules.py which reads RULES.md from the disk and prepends the whole thing to every prompt. That helps the rules to stop fading as context fills because it is reinforced every prompt.

Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better. Emptying the memory helps too to avoid Claude making up stuff that messes with how I want it to act.

The general gist of inject_rules.py is :

RULES_PATH points at RULES.md

reads it with encoding='utf-8-sig' so the BOM is stripped

wraps it in a JSON object — hookSpecificOutput.hookEventName = "UserPromptSubmit", additionalContext = a preamble plus the full rules text

the preamble is the line you see above the rules: rules are in force for this turn, run rule 33's five tests before raising anything unasked

prints that JSON to stdout, which is how Claude Code takes it in

on OSError it returns 0 silently — if RULES.md is missing or unreadable, nothing is injected and the turn proceeds with no rules

Re: Handbook.md shows that long policy documents do not reliably govern agents

#97

Post by QuercusMax »

pmarreck wrote: post #28
mcdeltat wrote: post #11

Earlier quoted context omitted.

Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?

You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass. In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(…

I've seen pretty much every model including Claude bypass commit hooks constantly if there's even a tiny bit of friction, instead of pausing and asking me to help fix why it can't run the hooks (usually something that needs to be done outside the sandbox like `npm ci`).

That's why it's critical to have these checks run in a context that the model can't bypass, such as via github actions that block PR merges. Generally works pretty well for stuff at work (where we have all this stuff set up as part of continuous integration checks), but it means that for personal stuff I have to set up quite a bit more infrastructure to ensure that models don't just skip running tests (which they LOVE to do).

Re: Handbook.md shows that long policy documents do not reliably govern agents

#98

Post by dluxem »

Muromec wrote: post #58
pmarreck wrote: post #28

Earlier quoted context omitted.

You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass. In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(…

>In this case, it is an advisory control only, because the LLM can also unhook that hook you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.

This appears to be what the claude code-review is doing:

https://github.com/anthropics/claude-code/blob/7ef6eec9d9ba8...

Re: Handbook.md shows that long policy documents do not reliably govern agents

#99

Post by tedggh »

_fat_santa wrote: post #25
DiabloD3 wrote: post #6

This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that. Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely cont…

At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models. Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"

Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#100

Post by kridsdale1 »

batshit_beaver wrote: post #83
wongarsu wrote: post #26

Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; re…

The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly gover…

Models learn. It just costs $10B and 1 year to do what a human does every night.
Post reply on HN