Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Handbook.md shows that long policy documents do not reliably govern agents

arxiv.org

Post reply on HN
Print view View original

111–120 of 237 posts

Re: Handbook.md shows that long policy documents do not reliably govern agents

#111

Post by horsawlarway »

tedggh wrote: post #99
_fat_santa wrote: post #25

Earlier quoted context omitted.

At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models. Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"

Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.

I'm not the above poster, but my experiences generally line up with theirs.

The first 20% of context is usually very solid. Agent stays on task, is responsive and focused on the original goal.

The last 20% of context is usually pretty bad. Agent tends to loop, repeat, drift away into sub-goals (or non-goals).

The middle 60% is a bit of a grab-bag. Sometimes it's fine, sometimes it's not. It varies by task, prompt, and luck.

The above 25% and 50% limits seem like reasonable and simple rules to keep agents in the productive zones of usage.

It's generally not worth doing extensive testing to try to find your exact threshold unless you're running exactly the same prompt at scale, on similar data.

If you are in that spot, I won't actually recommend langchain tooling (I've been somewhat frustrated with it) but I think the concepts are worth understanding.

You can do something like https://docs.langchain.com/oss/python/langchain/test/evals. Trajectory evals in particular seem to suffer as context length grows (https://docs.langchain.com/oss/python/langchain/test/evals#t...).

I've used this for testing an agent that handles provisioning and permissions for developers with SaaS tooling (ex - if a developer asks, go provision an account [saml/oauth/etc] with the 3rd party, configure access, and let them know how to use it).

I'm about 6 months out of date at this point, so I won't speak for the latest models, but what was available early this year absolutely suffers from extended context lengths.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#112

Post by 8note »

tedggh wrote: post #99
_fat_santa wrote: post #25

Earlier quoted context omitted.

At my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models. Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"

Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.

250k because thats what the smaller models supported and theres better training data in that part of the window

there have been some papers suggesting that the useful context is even smaller, and stays fixed as you change the context window size.

as a more general case though, i think the possibilities for what youd need to include in training to have many different paths of text be well represented enough over the long window means the later tokens will almost always be a lot more random than early ones?

there's a sheer amount of bits problem.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#113

Post by Catloafdev »

thaumasiotes wrote: post #105
Catloafdev wrote: post #95

Earlier quoted context omitted.

> There are no "attention heads" Yes there quite literally is internally in an LLM.

https://bactra.org/notebooks/nn-attention-and-transformers.h... Just because something is called an "attention head" doesn't mean the terminology makes sense. If I'm reading the article right, hungryhobbit is accurately describing a single attention head. And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungr…

There quite literally are internals that are called attention heads in modern LLMs. Saying 'there are no attention heads' is objectively false, which definition you follow is irrelevant.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#114

Post by duncangh »

simonw has been correctly asserting that there is no deterministic way to prevent hallucinations. I’d argue that this obstinance could be the seed of a strong argument in favor of their capacity to eventually coalesce some form of consciousness

Re: Handbook.md shows that long policy documents do not reliably govern agents

#115

Post by Aurornis »

DiabloD3 wrote: post #6

This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that. Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely cont…

> Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.

This is just not true. Any local LLM you can host on consumer-accessible hardware has all of these defects, too. Adjusting the knobs doesn’t solve everything.

The closest you can get to frontier performance is Kimi K3, but you’re not hosting that unless your budget is on the order of a nice house in a good metro area.

I like my local LLMs as much as the next person and my office is currently uncomfortably warm from the amount of compute happening, but I would never agree that local LLMs solve all of the common LLM defects. This is peak wishful thinking.

In my experience, the local models and even the larger ones that we can’t run at home suffer more from long context degradation than the frontier models. You are exactly right that you need to manage context length, but even at fp16/bf16 the local models have a lower ceiling for usable context length in my experience.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#116

Post by missmoss »

This is real. I was very angry that AI kept breaking the rules I wrote, so I asked Claude to crawl its own history logs. And then I found: every time after it breaks a rule, the chance of breaking rules goes up.

I feel this is like few-shot in reverse. Few-shot is supposed to be good examples AI should follow. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.

I wanted to know if there is any difference between writing rules in the prompt, writing them in CLAUDE.md, or not writing at all. So I did some short tests before. I asked Claude to open brand new sessions, test different topics with the rules I want to apply. The result turns out to be: in a fresh session, no matter the rule is in a prompt or in CLAUDE.md, models (Opus 4.8, 5 or Fable) all follow it fine, across models. Even Opus 4.8, the one always violates rules in our conversations, does it well.

I suspect it's the long context that breaks rules. But simulating a long conversation experiment is kind of hard, I still haven't found a good way to test it. So seeing this paper now, it completely answers the question I was stuck on these few days.

Besides, something caught my eyes in this paper: sometimes the model does run the check by the rules, and it really finds the violation, but its narrative still insists on its original wrong output.

My current approach is same as everyone here: use a separate hook or post-check to fix things. Because if you let the model fix it during generation, its narrative or main generation part sometimes just rejects the rule error it found.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#117

Post by win311fwg »

msejas wrote: post #64
drob518 wrote: post #43

Earlier quoted context omitted.

What do you mean by a graph of one shot prompts?

Most people jump straight to agents when what they actually need is a graph. Example: a mining company receives free-text reports from field geologists. You could have: Geologist report -> LLM call extracts minerals we are looking for (you inject a db query result on the user prompt), locations, assay mentions and risks into structured fields -> LLM call classifies evidence into positive indicators, negative indicato…

I jump straight to graphs (unless it truly cannot be solved with a graph), but then the stakeholders get upset that I didn't use agents. Doesn't fit their marketing plan, I suppose. I don't mind dealing with that, but I can see why most jump to agents in order to avoid the human conflict.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#118

Post by Aurornis »

ux266478 wrote: post #87
satvikpendem wrote: post #79

Earlier quoted context omitted.

How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.

As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline? Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like tha…

> Well the answer is that VC-backed companies

Look, I enjoy local LLMs as much as anyone, but I think there is some motivated reasoning happening in this thread to try to make local LLMs sound like a utopia against those evil VCs.

Local LLMs suffer from the same problems.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#119

Post by cyanydeez »

haus20xx wrote: post #85
drob518 wrote: post #56

Earlier quoted context omitted.

Doesn’t that blow your token cache hit rate and balloon your costs (essentially everything is billed at the input token rate, not the cached rate)?

If it moved a block like => //Removed project_prompts //reinserted Cache would break but if you did instead //Duplicated with new user messsage It wouldn't bust cache, it would just make your input prompts slightly larger. Technically inefficient as you're duplicating the same rules over and over but I imagine for a smallish checklist/principles that it is tremendously more efficient than a cache break every message

so first, we treat context like a stack. Things are popped onto the stack; if we try to remove anything but the top of the stack, we'll break the cache.

So a good harness should be popping tool prompts, user hints after every return from the top so the context of tools and user hints never repeats and are allways prefixed to the current user prompt.

I've never inspected any harness, but opencode with the dynamic context compaction plugin appears to do this well.

Re: Handbook.md shows that long policy documents do not reliably govern agents

#120

Post by 8note »

wongarsu wrote: post #103
kridsdale1 wrote: post #100

Earlier quoted context omitted.

Models learn. It just costs $10B and 1 year to do what a human does every night.

LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models

does lora do a good job at teaching the model new things that werent in the training data?

without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with

Post reply on HN