Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust large context windows

garrit.xyz

Post reply on HN
Print view View original

201–210 of 211 posts

Re: Don't trust large context windows

#201

Post by saberience »

kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I never use Opus etc after 50% token usage (and from reading other devs blogs and on Twitter it seems this is a comment sentiment) because it falls off an intelligence cliff at that point.

I mean, I really, really see intelligence tank at a certain amount of context usage. I always start a new session when any implementation work is starting or when starting a new plan.

So I clean context before writing a plan, I clean context before any implementation of a plan. My first prompt is always putting enough of my own context, copy and pastes of docs, etc, to ensure the plan creation is good. Once the plan is made I clean the context and get Opus to implement said plan.

Out of all the methodologies I've tried, this seems to be the best in terms of output quality.

Re: Don't trust large context windows

#202

Post by orbital-decay »

dofm wrote: post #123

I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…

That's totally expected. The field of large-scale generalist AI is entirely novel and experimental. It doesn't and can't have rigor of more mature disciplines that had decades to develop.

That said, there's no cargo cult in blindly using heuristics for certain fundamental LLM phenomena that have tons of good studies backing them (e.g. have no extra distractors, group and delimit pieces of the context, etc). If you want quantitative rigor, perform correct evals on your specific task and model.

Re: Don't trust large context windows

#203

Post by ausbah »

deliciousturkey wrote: post #113

I dislike the non-specificity of "models" here. Different models have different attention architectures, and can therefore have significant differences in long-context behavior. It's true that long context is an issue can most models do drop off in quality, but I would not extrapolate behavior of old models to new ones.

could you explains on how? what changed with attention mechanisms to allow for such a shift?

Re: Don't trust large context windows

#204

Post by xnx »

dofm wrote: post #123

I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…

I use the word "superstition" to describe a lot of this.

Some of the explanation is that these systems are hard to understand and changing fast. Another part of the explanation is that AI has removed so much of the need for the expertise that people have that they contrive complications so they can feel they are still doing something.

Re: Don't trust large context windows

#205

Post by kevincox »

bob1029 wrote: post #40

I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…

> There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia

Makes me wonder if it would be best to have some sort of "fork" operation to start the new agent. Rather than starting from blank it inherits the existing context (which is already cached for evaluation) plus a bit on top for its specific task. Much like the system call there would essentially be two returns, the one in the agent says "You are the agent, perform the discussed work" and the parent gets the result produced by the agent.

Re: Don't trust large context windows

#206

Post by newterminus »

coldtea wrote: post #199
newterminus wrote: post #197

Earlier quoted context omitted.

it has a random seed cant you just set it

Wouldn't change much. Would just make the answer to the same exact prompt X repeatedly the same. It wouldn't change the fact that prompt X', functionaly indistinguisable from X, aside from small phrasing changes, can give a totally different looking answer.

when i make a statement describing the world it will be a totally opposite answer to the prompt "synthesize and analyze the steelman that X" because you are asking it to create something new and then you are giving it room to counter signal and some space to escape into but often it doesnt even countersignal much or at all in this analysis phase

Re: Don't trust large context windows

#207

Post by pavlus »

kevincox wrote: post #205
bob1029 wrote: post #40

I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…

> There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia Makes me wonder if it would be best to have some sort of "fork" operation to start the new agent. Rather than starting from blank it inherits the existing context (which is already cached for evaluation) plus a bit on top for its specific task. Much like the system call there would essentially be two returns, the one i…

I've actually tried something like that, with two tools:

* push("What I'm about to do"),

* pop("What I've achieved").

"Push" marks the position in current context after the call and returns "Proceed", "pop" erases everything after the matching call and replaces "Proceed" in it's result as what was passed to the argument to the "pop", effectively pruning long-winded head even inside one reasoning stream. In the end the model only sees how it decided to do work on something, and that it was already done, forgetting everything it between, except what it itself decided was important.

Gemma 4 31B QAT successfully uses it when navigating a maze, marking positions at intersections, exploring them, navigating back and pushing again if necessary. Smaller models often fail to mark positions and forget to backtrack as well, instead they try to rely on themselves to track their paths and navigate back (and also fail).

I think it should work for long-running deep research tasks, but I was too lazy to test it, because it all required a lot of code to glue this up, since most tools and libs are not designed to work like that, and now I'll need even more code to test it, without a purposeful task.

Re: Don't trust large context windows

#208

Post by krackers »

data-ottawa wrote: post #188
tgv wrote: post #172

Earlier quoted context omitted.

If you want my best guess: I think large context windows cannot be trained properly. There's not enough material, nor computing power, to train such large networks (to the same degree as small windows).

I feel this is a sort of inverse inspection paradox (the paradox that if you sample waiting time in a process, you’re more likely to sample a larger value). The LLM providers fine tune the models with some kind of information retrieval tasks, but to do so you must provide some non relevant context to bootstrap the session for the long context tasks. It would be very easy to do this in ways that train the sequence mod…

> (the paradox that if you sample waiting time in a process, you’re more likely to sample a larger value).

The deepseek v4 paper talks about one variant of this (related to failures) and how they mitigate it.

>During preemption, we pause the inference engine and save the KV cache of unfinished requests. Upon resumption, we use the persisted WALs and saved KV cache to continue decoding. Even when a fatal hardware error occurs, we can re-run the prefill phase using the persisted tokens in WAL to reconstruct the KV cache.

>Importantly, it is mathematically incorrect to regenerate unfinished requests from scratch, as this introduces length bias. Because shorter responses are more likely to survive interruption, regenerating from scratch makes the model more prone to producing shorter sequences whenever an interruption occurs. If the inference stack is batch-invariant and deterministic, this correctness issue could also be addressed by regenerating with a consistent seed for the pseudorandom number generator used in the sampler. However, this approach still incurs the extra cost of re-running the decoding phase, making it far less efficient than our token-granular WAL method.

Re: Don't trust large context windows

#210

Post by wrs »

KronisLV wrote: post #189
wrs wrote: post #168

Earlier quoted context omitted.

You can also just ask it to do work in a subagent. It will write a plan and launch the subagent to do the actual code, keeping it out of the main context. In addition, you can co-author a plan for a biggish chunk of work, divided into stages, have it launch a subagent for phase 1 and check its work, then ESC-ESC to go back to just after you wrote the plan and have it do phase 2. Repeat until done. This keeps the over…

My problem with regular sub-agents is that after around 2-4 hours the main agent stops working on a task and asks for user input no matter how I tell it to continue autonomously until the ~5-15 stage plan is done, when when it has a clear plan that's made with the plan mode and instructions to continue autonomously. It's happened multiple times where I give it a task before going to sleep and when I come back it's st…

Well, 2-4 hours of autonomous work is outside my comfort zone to start with. But have you tried Claude Code’s “auto mode”? I haven’t seen this premature stopping since it was introduced.
Post reply on HN