Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust large context windows

garrit.xyz

Post reply on HN
Print view View original

21–30 of 211 posts

Re: Don't trust large context windows

#21

Post by Bolwin »

kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer

Re: Don't trust large context windows

#22

Post by cyanydeez »

kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

As the gamblers say at the poker table: If you can't figure out who the mark is when you site down...

Re: Don't trust large context windows

#24

Post by cyanydeez »

hypfer wrote: post #13
mg wrote: post #9

Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an on…

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

come on now, we can't just not escape the permanent underclass by using our brains, we've also got to use up all the resources while doing it.

Re: Don't trust large context windows

#25

Post by eterm »

Bolwin wrote: post #21
kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer

60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md.

60k isn't much bigger than the system prompt.

Re: Don't trust large context windows

#28

Post by wg0 »

Bolwin wrote: post #21
kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer

Not specific to Opus but yes it would make mistakes. I usually try to keep context window under 10%

Re: Don't trust large context windows

#29

Post by properbrew »

Bolwin wrote: post #21
kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer

I hate to do the "you're holding it wrong" trope, but I think you might have something misconfigured somewhere unless you missed a 0, because just past 60k tokens is such a small context window to be seeing issue in.

Do you have any old documentation that it's picking up and referencing? If you set all claude settings back to default do you see the same issue?

Re: Don't trust large context windows

#30

Post by danielbln »

eterm wrote: post #25
Bolwin wrote: post #21

Earlier quoted context omitted.

I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer

60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md. 60k isn't much bigger than the system prompt.

Yeah 60k is ludicrous, I've barely seeded the context at that point and I don't see context related degradation until well into the 600-700k.
Post reply on HN