101–110 of 211 posts
Can't speak to how good those tests are, but they can't be worse than anecdotal evidence for something as vague/subjective as LLM performance.
Earlier quoted context omitted.
In this thread: People tossing coins independently and fighting over the result they got.
No it's not. It seems that people have different workflows or repos, or memories or prompts or expectations.
Also, some colleagues were playing around with RTK (https://github.com/rtk-ai/rtk), which decreases the amount of token used by tool calls and, although it seems an interesting idea, I am pretty sure there are many caveats. Although, I believe if these type of tools prove to be efficient enough, perhaps harnesses will have them natively.
Almost every comment here is appealing to personal experience. By contrast, OP refers to two studies that compare performance on some kind of standardised test over a range of models. Can't speak to how good those tests are, but they can't be worse than anecdotal evidence for something as vague/subjective as LLM performance.
I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller. I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting…
I see it pretty frequently in troubleshooting and data analysis flows where it will dump the data collection and aggregation into a sub agent then pull out a summarized result.
I'll do something similar where I have the main agent maintain context in a design doc/markdown file and update as it goes along. Then I can clear/restart/handoff at will
Earlier quoted context omitted.
60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md. 60k isn't much bigger than the system prompt.
Yeah 60k is ludicrous, I've barely seeded the context at that point and I don't see context related degradation until well into the 600-700k.
I routinely get claude to do things pretty decently and finish up easily in the 4-5 digit range of tokens. It seems to be doing the right kind of thing to not waste its time looking at 1000 files.
This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…
I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer
Frequently it'll be something as small as 1 sentence of a prompt many messages ago.
When cases like that happen, I reset the context and try to be explicit about assumptions and requirements to keep it off the "tainted" path. Other times it's actually useful and agents will do things they normally wouldn't do once the state is tainted. For instance, if you're testing a chat bot's ability to stay on topic, you can seed the context early with what you want it to do. It generally will refuse initially but later on in the conversation it will still silently take that seeded context into account almost "subconsciously" and become more likely to do the thing it originally refused.
If you don't point out what's wrong I find the LLM will go into great technical detail which consumes a lot of tokens, but not 'see the wood for the trees'.
It seems to me human beings also have mechanisms to compact context, which may be why we can forget what we came into a room for when going through doorways. I think it would be interesting to research which markers we use to compartmentalize our thinking.