Hi I just read your methodology and I have a quick question about how the handbook are parsed and feed into the context window? The article mentioned that each handbook contains roughly 8K to 79K tokens of extracted text, and did the harness system use grep or search tools to find relevant chunks and feeds to the context, or did it just feeds all the pdf output to the model? There might by distribution bias between real world tasks e.g. Agents grep keywords from docs and only use the relevant chunks. So all the models of the overall pass@1 is relatively low compared to real world scenarios, that might not be the same precision that user experience when they actually handle the daily task? How did the benchmark bridge the gap?
Handbook.md shows that long policy documents do not reliably govern agents
231–237 of 237 posts
Re: Handbook.md shows that long policy documents do not reliably govern agents
#232Earlier quoted context omitted.
This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.
Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
No because the CLAUDE.md is included in the first prompt which is given priority in the context. On prompt 20, it will remember message 1 better than message 12.
Re: Handbook.md shows that long policy documents do not reliably govern agents
#233In my experience, the more structure you enforce on models, the worse they perform and the less they actually do what you want.
yep all the advice about context engineering, harness whatever is so silly. ai doesnt give a flying fuck about some IMPORTANT instruction in your claude.md. It does what its has been trained to do. So find out what its trained to do and just use it to do that. This is not general intelligence.
Yes. And all the “context engineering”, document/decision frameworks etc just make it way harder to work with the ai. You put in a lot of work just for an illusion of control.
Re: Handbook.md shows that long policy documents do not reliably govern agents
#234Earlier quoted context omitted.
Are you the Deepclause author? I tried it yesterday and my first reaction was that it was slow. Perhaps I’m doing something wrong, however. Running against the same model in Pi was lightspeed in comparison. Second reaction is the prompt editor needs to handle more than a single line of text and it needs Emacs editing key bindings like Pi. I’m happy to do some testing on it and provide further feedback. What’s the bes…
Yes, I am the author and thanks so much for trying! Please do submit a github issue. My first suspicion about the speed is that maybe an inner loop is taking too many turns until the model finally realizes that a task is finished (so that in turn the runtime knows whether the predicate failed or not and can continue execution accordingly). Happy to take a closer look! The point about multiline prompts is very valid o…
Cool, will do. Was playing around with it last night some more.
Re: Handbook.md shows that long policy documents do not reliably govern agents
#235For Claude, I used a UserPromptSubmit hook running inject_rules.py which reads RULES.md from the disk and prepends the whole thing to every prompt. That helps the rules to stop fading as context fills because it is reinforced every prompt. Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better…
why this vs having some response hook check the next outputs against the bunch of rules, and injects only when it goes off track?
Just the first thing I thought of to reinforce the rules. I don't use all my tokens anyway (now), so for me, there's no reason to introduce a point of failure, just reinforce every time.
Re: Handbook.md shows that long policy documents do not reliably govern agents
#236[flagged]
Re: Handbook.md shows that long policy documents do not reliably govern agents
#237[flagged]