Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust large context windows

garrit.xyz

Post reply on HN
Print view View original

31–40 of 211 posts

Re: Don't trust large context windows

#31

Post by mg »

hypfer wrote: post #13
mg wrote: post #9

Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an on…

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

It is the other way round.

In an interactive session, adding "Fine, but make the button red" after the model generated a first solution more than doubles the tokens used. As the model now not only gets the original code and the feature request but also the updated code plus the change request as input tokens.

Sending a feature request to an LLM and then sending the feature request again with "The button shall be red" only doubles the tokens used.

Re: Don't trust large context windows

#33

Post by redox99 »

hypfer wrote: post #13
mg wrote: post #9

Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an on…

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

Probably like 1% of the energy an average person spends on driving.

Re: Don't trust large context windows

#34

Post by da_grift_shift »

eterm wrote: post #25
Bolwin wrote: post #21

Earlier quoted context omitted.

I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer

60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md. 60k isn't much bigger than the system prompt.

>you might have some false memories or incorrect instructions in your CLAUDE.md

    "YOU'RE HOLDING IT WRONG!"

Re: Don't trust large context windows

#35

Post by csomar »

kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I have a custom build command for a rust project (yarn build:lib) and my experience is 120k for GLM and roughly 200-300k for Opus. After that, they default to cargo build.

Re: Don't trust large context windows

#36

Post by stavros »

fullstackchris wrote: post #18
kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

Thats another problem of this post, the author mentions Claude but not explicitely what models... 100k tokens "by lunch" is also not my finding, the newer models will hit that already right in the initial exploratory phase

I found "by lunch" odd too, but considering that Claude wrote the article, it's not going to know specifics.

Re: Don't trust large context windows

#37

Post by da_grift_shift »

kristianc wrote: post #16
nopurpose wrote: post #11

Earlier quoted context omitted.

Is it adhoc or you use more structured approaches like openspec? I also tend to work on a plan first, but it stays as in-session todo, which is hard to reference later.

It's ad hoc / my own framework, just found something which works for me. The exact structure is - Work Mode - HITL/AFK - Problem Statement - Who It Affects - Primary / Secondary User - User Stories - Business Case - Why Now - Success Critera - In Scope/Out of Scope [Out of Scope v. important) - Thinnest Slice (This I've found super valuable, means you max out the amount of 'product' for your buck and avoid diminishin…

Is there back-and-forth? How long do these get? Can you share an example?

Re: Don't trust large context windows

#38

Post by trapexit »

csomar wrote: post #35
kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I have a custom build command for a rust project (yarn build:lib) and my experience is 120k for GLM and roughly 200-300k for Opus. After that, they default to cargo build.

My projects have specific build/verify steps as well, and after a certain point Claude forgets to run them. I’m going to try a “No brown M&Ms” hook to halt Claude if it tries to run the default command instead of the instructed commands from CLAUDE.md. Perhaps this will be a good signal that a compacted or fresh session is needed at that point to avoid mistakes.

Re: Don't trust large context windows

#39

Post by Raphael_Amiard »

redox99 wrote: post #33
hypfer wrote: post #13

Earlier quoted context omitted.

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

Probably like 1% of the energy an average person spends on driving.

Average american is what you mean

Re: Don't trust large context windows

#40

Post by bob1029 »

I've been able to avoid context size issues by applying one simple constraint to my agent loop. What I do is prevent all tool calling in the user's top-level conversation thread. Anything that needs to tool call must happen in a recursive invoke of the agent, which returns whatever results to caller.

I can keep the same high level conversation going for an entire day over a million LOC+ codebase without ever hitting meaningful token limits. No compaction or summarization tricks needed. I can burn 50 million tokens in recursive calls and still not touch 100k tokens in my root conversation thread.

There is some rework needed to "bootstrap" the agent each time it has to descend back into Narnia, but this is still far more efficient than carrying around one big flat context that tries to cover everything all the time.

Recursion is very effective at controlling token use, but it can only go so far. I've not observed any uplift for recursive depth beyond 1. I have seen the agent attempt it a few times, but the practical performance is simply not there. External symbolic recursion does not appear to be something the frontier models have been trained for. They are fantastic at emulating recursion in context, but we don't want that if we are trying to achieve a reduction in token use.

Post reply on HN