[flagged]
Don't trust large context windows
111–120 of 211 posts
Re: Don't trust large context windows
#112Re: Don't trust large context windows
#113I dislike the non-specificity of "models" here. Different models have different attention architectures, and can therefore have significant differences in long-context behavior. It's true that long context is an issue can most models do drop off in quality, but I would not extrapolate behavior of old models to new ones.
Re: Don't trust large context windows
#114Almost every comment here is appealing to personal experience. By contrast, OP refers to two studies that compare performance on some kind of standardised test over a range of models. Can't speak to how good those tests are, but they can't be worse than anecdotal evidence for something as vague/subjective as LLM performance.
I'll respond with more anecdotal evidence, the Llama family has been terrible at following directions in all the tests I've done--not sure about the other models in RULER.
In the Chroma results, they look at Sonnet 4 which was also terrible in my experience. The same prompt that worked perfectly in Sonnet 4.5 would fail miserably in Sonnet 4
Would be good to see newer tests with both SOTA and open weight. The SOTA ones always seem to follow directions and stay on topic better but it'd be good to have some data to back it up.
Re: Don't trust large context windows
#115Funny to read about that superpowers repo, since only yesterday I wrote skills to do some markdown-plan centered aproach. I feel like smallish local models are getting capable of lots of things now, but they need lots of structure for resiliency.
Re: Don't trust large context windows
#116[dead]
Re: Don't trust large context windows
#117Earlier quoted context omitted.
I have a custom build command for a rust project (yarn build:lib) and my experience is 120k for GLM and roughly 200-300k for Opus. After that, they default to cargo build.
My projects have specific build/verify steps as well, and after a certain point Claude forgets to run them. I’m going to try a “No brown M&Ms” hook to halt Claude if it tries to run the default command instead of the instructed commands from CLAUDE.md. Perhaps this will be a good signal that a compacted or fresh session is needed at that point to avoid mistakes.
I mean, that’s basically the magic of the harness. The whole thing that skyrocketted the intelligence is that the harness (cli tool) prevent the LLM from editing the file before reading it.
Can you imagine even a junior making such a mistake?
Re: Don't trust large context windows
#118Funny to read about that superpowers repo, since only yesterday I wrote skills to do some markdown-plan centered aproach. I feel like smallish local models are getting capable of lots of things now, but they need lots of structure for resiliency.
Yeah I’ve been using gpt-5.3-codex-spark in Codex lately and it can be surprisingly good and it’s super fast. However it needs more explicit instructions.
Re: Don't trust large context windows
#119I think of the context window as a pot of soup that you add ingredients to between meals. If you have a relatively focused recipe and you are able to add only the ingredients you want, the soup stays good. If you or the agent add an ingredient that isn't fresh, it is going to be difficult to salvage and it is better to start over with a new pot.
It is not that agents can't function with a large context window, they can if that information generally has a desirable signal (like a large initial document or a well-focused session). Mistakes and the confusing signals that come out of fixing mistakes are why performance degrades. I start to trust the context window less not as a matter of size but the amount of friction we run into. The friction can be random but it is more often an issue with the path that I have us on.
Re: Don't trust large context windows
#120I think of the context window as a pot of soup that you add ingredients to between meals. If you have a relatively focused recipe and you are able to add only the ingredients you want, the soup stays good. If you or the agent add an ingredient that isn't fresh, it is going to be difficult to salvage and it is better to start over with a new pot. It is not that agents can't function with a large context window, they c…
Hmm iirc if you ask Claude it itself recommends one conversation per task.
That’s what I did intuitively anyway.