Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust large context windows

garrit.xyz

Post reply on HN
Print view View original

81–90 of 211 posts

Re: Don't trust large context windows

#81

Post by KronisLV »

bob1029 wrote: post #40

I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…

For anyone using Claude Code, ask it to do all the work in workflows (it has a tool for that), they released that feature together with Opus 4.8 and it also seems a bit better at doing long tasks as well. The main conversation just orchestrates the work at that point.

Re: Don't trust large context windows

#82

Post by SeriousM »

magicalhippo wrote: post #53
kristianc wrote: post #16

Earlier quoted context omitted.

It's ad hoc / my own framework, just found something which works for me. The exact structure is - Work Mode - HITL/AFK - Problem Statement - Who It Affects - Primary / Secondary User - User Stories - Business Case - Why Now - Success Critera - In Scope/Out of Scope [Out of Scope v. important) - Thinnest Slice (This I've found super valuable, means you max out the amount of 'product' for your buck and avoid diminishin…

I guess I've stumbled into something similar. Though I don't have a fixed format like yours. I first do a lot of back and forth to generate what I call a design document also includes rationales for various points or decisions. I use both Claude and Codex to iterate on this until I'm happy. The end result includes a lot of what you mention. I then start a fresh conversation, make it analyze the design document and co…

I found myself in a similar workflow. Depending on the task at hand (starting a new project, enhancement, maintenance), I let the agent create/read the markdown files that I keep updated (AGENT, STATE, ROADMAP, DESIGN, ARCHITECTURE, (CODESTYLE if I plan to modi it myself)). Then I choose the various roles that I need in this session and and have a planning phase. After that, the agent is starting implement the changes and I have a manual correction phase.

This flow works for my needs, building idea demos, prototypes or tools for my own sake. I don't let agent code in our main code base where everything is still hand tailored. That's a conscious decision.

I noticed that the cheaper models (flash, ...) are quite hard to hold back changing files. A question for possible options sometimes results in "yes, I'll go with option A" without asking back. Frontier models on the other hand love to plan and ask you deliberately for your consent.

I use pi.dev with almost no skills at all to understand how models really work and "feel" to work with.

Re: Don't trust large context windows

#84

Post by Muromec »

bob1029 wrote: post #40

I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…

I have a different way, but still trying to figure out how well it works. Instead of going into recursion, the agent is allowed to restart the thread by doing the summarize/debrief/reflect pass, writing key findings into persistent memory and rewriting the prompt whenever the context goes too large or it gets stuck. Recursion with TCO if you may.

In a way it's a generalization of the spec-driver approach, but in addition to the the formal spec the carryover buffer lives in the memory.

Re: Don't trust large context windows

#85

Post by throwaway314155 »

arcanemachiner wrote: post #19
kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

Opus 4.6 was on drugs past 200k, I skipped 4.7, 4.8 did good up to ~350k, and Fable did great beyond 400k, in my limited testing. The quality does appear to be trending upwards.

> Opus 4.6 was on drugs past 200k

Which drugs?

Re: Don't trust large context windows

#86

Post by CjHuber »

embedding-shape wrote: post #66
CjHuber wrote: post #50

Earlier quoted context omitted.

I'm always a bit confused when people say things like this. 60k token is often more than the initial context I feed the model with. And I don't think I ever had a productive session that began under 150k tokens.

Bit of what makes it so fun, our experiences seem to wildly differ! On one hand, you have experiences like yours, but then my own experience is that I never had a productive session when the scope grows beyond 150K tokens! If I needed 60K just as a starting context, I'd take that to mean the suggested change is way to large, and if the model cannot solve the entire thing within maybe 15-20% of the total context size,…

Yeah indeed it's very interesting. And the 60k initial context don't even contain the suggested change yet. For me if I don't do this the current models tend to fixate and local patches instead of tracing symbols and making a holistic model of what a change interacts with in the codebase

Re: Don't trust large context windows

#88

Post by stogot »

Muromec wrote: post #84
bob1029 wrote: post #40

I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…

I have a different way, but still trying to figure out how well it works. Instead of going into recursion, the agent is allowed to restart the thread by doing the summarize/debrief/reflect pass, writing key findings into persistent memory and rewriting the prompt whenever the context goes too large or it gets stuck. Recursion with TCO if you may. In a way it's a generalization of the spec-driver approach, but in addi…

Kiro does this automatically from what I can tell using it

Re: Don't trust large context windows

#89

Post by justinclift »

throwaway314155 wrote: post #85
arcanemachiner wrote: post #19

Earlier quoted context omitted.

Opus 4.6 was on drugs past 200k, I skipped 4.7, 4.8 did good up to ~350k, and Fable did great beyond 400k, in my limited testing. The quality does appear to be trending upwards.

> Opus 4.6 was on drugs past 200k Which drugs?

The way it hallucinates stuff, it'd probably be something in the LSD family. ;)

Re: Don't trust large context windows

#90

Post by justinclift »

endless1234 wrote: post #55
SwellJoe wrote: post #47

Opus in recent versions is fine beyond 100k, but I usually do try to keep it under 200k. But, this is also why so-called "memory" systems are usually a mistake that make the models dumber. They don't have memory, they only have context, and every irrelevant fact you shove into the context is less context for the problem. Less distractions, better results. The way to have the agent remember things is to have it docume…

At least for me, Opus keeps writing stuff to memories, only to consistently forget checking those memories before doing the same mistake again. This ("remember to check memories!") is of course then again written as a memory... Clearly not a very well working system, yep.

I've explicitly banned Opus from creating memories unprompted, as it would often save info that's incorrect and which would then be propagated to future sessions until caught. Ugh x 10.
Post reply on HN