Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust large context windows

garrit.xyz

Post reply on HN
Print view View original

51–60 of 211 posts

Re: Don't trust large context windows

#51

Post by mg »

jgilias wrote: post #42
mg wrote: post #31

Earlier quoted context omitted.

It is the other way round. In an interactive session, adding "Fine, but make the button red" after the model generated a first solution more than doubles the tokens used. As the model now not only gets the original code and the feature request but also the updated code plus the change request as input tokens. Sending a feature request to an LLM and then sending the feature request again with "The button shall be red"…

The cost is far from linear though. Because of prompt caching and the fact that generally output tokens are a lot more expensive than input tokens.

Agreed that it is not linear.

I wrote my own agent, and it sends data to LLMs in this order: "General Prompts (How to write good code)" + "The Code" + "The Feature Request". This means the KV cache will be used even when the feature request changes.

And output tokens are usually way less than the input tokens.

So I think that my approach is very lightweight on token usage compared to an interactive session.

It would be interesting to measure it for the other agents out there. Sending a feature request two times vs an interactive session.

Re: Don't trust large context windows

#52

Post by steveridout »

I wonder how much this depends on the quality and consistency of the context?

For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental.

Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?

Re: Don't trust large context windows

#53

Post by magicalhippo »

kristianc wrote: post #16
nopurpose wrote: post #11

Earlier quoted context omitted.

Is it adhoc or you use more structured approaches like openspec? I also tend to work on a plan first, but it stays as in-session todo, which is hard to reference later.

It's ad hoc / my own framework, just found something which works for me. The exact structure is - Work Mode - HITL/AFK - Problem Statement - Who It Affects - Primary / Secondary User - User Stories - Business Case - Why Now - Success Critera - In Scope/Out of Scope [Out of Scope v. important) - Thinnest Slice (This I've found super valuable, means you max out the amount of 'product' for your buck and avoid diminishin…

I guess I've stumbled into something similar. Though I don't have a fixed format like yours. I first do a lot of back and forth to generate what I call a design document also includes rationales for various points or decisions. I use both Claude and Codex to iterate on this until I'm happy. The end result includes a lot of what you mention.

I then start a fresh conversation, make it analyze the design document and code, and for larger changes, generate a high-level implementation document which includes concrete phases or steps. I review this plan and iterate if necessary.

Then for each phase I make it generate a detailed plan for that phase and save it along side the other documents. Once the phase is over, I make it write a summary of what was done, decisions made and reasons for it. And typically a good point to compact the model's context.

These documents gives additional context for when I make another model do code review, and help illuminate drift or gaps from the main design document.

Re: Don't trust large context windows

#54

Post by perching_aix »

mg wrote: post #9

Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an on…

Which model is leading the pack for you?

Re: Don't trust large context windows

#55

Post by endless1234 »

SwellJoe wrote: post #47

Opus in recent versions is fine beyond 100k, but I usually do try to keep it under 200k. But, this is also why so-called "memory" systems are usually a mistake that make the models dumber. They don't have memory, they only have context, and every irrelevant fact you shove into the context is less context for the problem. Less distractions, better results. The way to have the agent remember things is to have it docume…

At least for me, Opus keeps writing stuff to memories, only to consistently forget checking those memories before doing the same mistake again. This ("remember to check memories!") is of course then again written as a memory... Clearly not a very well working system, yep.

Re: Don't trust large context windows

#56

Post by perching_aix »

da_grift_shift wrote: post #34
eterm wrote: post #25

Earlier quoted context omitted.

60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md. 60k isn't much bigger than the system prompt.

>you might have some false memories or incorrect instructions in your CLAUDE.md "YOU'RE HOLDING IT WRONG!"

[dead]

Re: Don't trust large context windows

#58

Post by wood_spirit »

Yes context management is key.

I do my own framework and spend a lot of time trying to debug this and it’s not so much the context size in hard numbers but rather the probability that there is debris or wrong directions in the window that are drowning out the things the user thinks are important.

This manifests in the llm that keeps going back to doing the thing that failed when they tried it just before the last approach etc. The frequency of things in the context window give weight even if they are the wrong things.

I have a lot of tricks like not giving the llm lots of tools but rather giving it a tool it can use to search for tools etc.

But the bigger solution is in process where you use something like superpowers to force the llm through stages and you control the context that carries forward.

Re: Don't trust large context windows

#59

Post by wood_spirit »

endless1234 wrote: post #55
SwellJoe wrote: post #47

Opus in recent versions is fine beyond 100k, but I usually do try to keep it under 200k. But, this is also why so-called "memory" systems are usually a mistake that make the models dumber. They don't have memory, they only have context, and every irrelevant fact you shove into the context is less context for the problem. Less distractions, better results. The way to have the agent remember things is to have it docume…

At least for me, Opus keeps writing stuff to memories, only to consistently forget checking those memories before doing the same mistake again. This ("remember to check memories!") is of course then again written as a memory... Clearly not a very well working system, yep.

In my own multi agent framework I use cheap models to check the responses of the expensive models, as well as using multiple expensive models adversarially in debate. The cheap models are great at spotting eg the model getting stuck in the alternate between two broken ideas or not following code conventions or missing a step in the skill and so on. I’m currently working on making them detect user corrections and police that going forward to intervene when the expensive models forget the thing you just corrected them about etc.

Re: Don't trust large context windows

#60

Post by Etheryte »

bob1029 wrote: post #40

I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…

How do you get the agent to stick to it without constantly rejecting tool calls with the same description? I've tried a similar setup a number of times and it tends to forget about this constraint very quickly.
Post reply on HN