Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Handbook.md shows that long policy documents do not reliably govern agents

arxiv.org

Post reply on HN
Print view View original

101–110 of 237 posts

Re: Handbook.md shows that long policy documents do not reliably govern agents

#101

Post by smu3l »

nonethewiser wrote: post #59
mcdeltat wrote: post #3

Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before. I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things i…

This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.

Conceptually the same thing though. Claude.md is sourced at the beginning of the session, so will be pretty far back in context, just like user prompts from the beginning of the session.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#103

Post by wongarsu »

kridsdale1 wrote: post #100
batshit_beaver wrote: post #83

Earlier quoted context omitted.

The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly gover…

Models learn. It just costs $10B and 1 year to do what a human does every night.

LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models

Re: Handbook.md shows that long policy documents do not reliably govern agents

#104

Post by alienbaby »

hungryhobbit wrote: post #88
eurekin wrote: post #20

Earlier quoted context omitted.

The needle benchmarks show, that models extended context works for the part, that can be explained as: "I can access/adress that part of the input". I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things…

There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly. After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ... ... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then…

Llm's have come on quite a bit beyond the fundamentals nowadays. Attention heads really are a thing.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#105

Post by thaumasiotes »

Catloafdev wrote: post #95
hungryhobbit wrote: post #88

Earlier quoted context omitted.

There are no "attention heads" or fixed number of things a model can pay attention to ... or at least not exactly. After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ... ... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then…

> There are no "attention heads" Yes there quite literally is internally in an LLM.

https://bactra.org/notebooks/nn-attention-and-transformers.h...

Just because something is called an "attention head" doesn't mean the terminology makes sense.

If I'm reading the article right, hungryhobbit is accurately describing a single attention head.

And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.

In particular, he is clearly responding to this clause in the comment above:

>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time

Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#106

Post by TiccyRobby »

ux266478 wrote: post #87
satvikpendem wrote: post #79

Earlier quoted context omitted.

How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.

As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline? Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like tha…

And what is the difference between cloud modeld vs local inference regarding this? It is still unclear in this thread - asking out of curiosity.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#107

Post by Analemma_ »

tedggh wrote: post #99
_fat_santa wrote: post #25

Earlier quoted context omitted.

At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models. Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"

Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.

In my own experience (both Claude and Gemini 1M), you start running into problems around 60-70%. A 25% ceiling sounds like total cargo culting to me.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#108

Post by regularfry »

Control vectors for the win? It feels like the way to fix this is to pull out a control vector immediately after processing the handbook and use that to steer later inference. That should stop the drift over long distances, but it doesn't guard against the handbook itself being too big.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#110

Post by pie_flavor »

nonethewiser wrote: post #59
mcdeltat wrote: post #3

Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before. I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things i…

This is not what the article is talking about. Its talking about policy documents not it forgetting something 5 prompts ago. In fact you adding things to CLAUDE.md is more what its talking about.

Claude is a next-token predictor, appending to a long text document. Prompts aren't an independent kind of thing from policy documents. It's all text in the backscroll.
Post reply on HN