Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Handbook.md shows that long policy documents do not reliably govern agents

arxiv.org

Post reply on HN
Print view View original

51–60 of 237 posts

Re: Handbook.md shows that long policy documents do not reliably govern agents

#51

Post by barumrho »

DiabloD3 wrote: post #6

This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that. Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely cont…

Would you mind sharing some of your setup? Which model and which params do you tweak (e.g. temperature)?

Re: Handbook.md shows that long policy documents do not reliably govern agents

#52

Post by JSR_FDED »

supermatt wrote: post #18

There was an article a few years ago called "Lost in the Middle: How Language Models Use Long Contexts" https://arxiv.org/abs/2307.03172 From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"

Richard Hendricks solved this decisively with middle-out compression

Re: Handbook.md shows that long policy documents do not reliably govern agents

#54

Post by hedgehog »

DanHulton wrote: post #47
simpaticoder wrote: post #13

Earlier quoted context omitted.

>Want it to go away, almost like magic? Local inference. Ah yes, magic that costs the same as a new car.

Not necessarily! If you have a semi-recent MacBook with even 32GB, you can run 20GB models that are pretty damn smart, with room to spare for the rest of your toolchain. If you’re reasonably connected to the code you’re writing and prompting the AI at the level of the code, not the level of the feature, you can get some fantastic results. Sure, it’s not the completely automated dreamland that’s been sold, but it’s st…

What models have you found to work for which tasks? I find the local models very useful but not in a way that's replaced cloud models (yet, I remain hopeful).

Re: Handbook.md shows that long policy documents do not reliably govern agents

#56

Post by drob518 »

Supermancho wrote: post #30
Muromec wrote: post #24

Earlier quoted context omitted.

Don't use the default harness, write your own instead.

This is the way. Making your own agent to have a sticky memory context that is prepended to every execution is necessary to ensure each task is bounded by those precepts.

Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?

Re: Handbook.md shows that long policy documents do not reliably govern agents

#58

Post by Muromec »

pmarreck wrote: post #28
mcdeltat wrote: post #11

Earlier quoted context omitted.

Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?

You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass. In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(…

>In this case, it is an advisory control only, because the LLM can also unhook that hook

you can in principle make a hook in the harness itself that will run an auditor prompt that checks it adhered to the policy, correct the model and also make it known in advance.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#59

Post by nonethewiser »

mcdeltat wrote: post #3

Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before. I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things i…

This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#60

Post by PunchyHamster »

pmarreck wrote: post #28
mcdeltat wrote: post #11

Earlier quoted context omitted.

Ok so what is the correct way to tell it "I don't care what is happening, you must uphold these rules at all times"? If it's not any configuration of .md files?

You need to make the rule concrete somehow. I call it a "control". So for example, instead of instructing it "always run tests before committing", you (or you have it) make a git commit hook that always runs the tests first and that refuses the commit if they don't pass. In this case, it is an advisory control only, because the LLM can also unhook that hook. And of course, it could also just disable the failing test(…

So you are essentially saying "you can't, you can only safeguard from effects of LLM eventually ignoring it"
Post reply on HN