Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust large context windows

garrit.xyz

Post reply on HN
Print view View original

11–20 of 211 posts

Re: Don't trust large context windows

#11

Post by nopurpose »

kristianc wrote: post #6

I'm getting a lot of mileage out of basically acting like the AI's Product Manager, and insisting that it writes up short PRDs for every feature we propose to build. That gives it a reference over time of everything that has been built, but also makes it less liable to drift with each one. Each one gets its own conversation. For me this is a happy medium between stopping it going off the rails but also making sure it…

Is it adhoc or you use more structured approaches like openspec? I also tend to work on a plan first, but it stays as in-session todo, which is hard to reference later.

Re: Don't trust large context windows

#12

Post by PeterStuer »

I've had no problem with Claude Code Opus 4.8 effort max using 20% token context (200k) on software development tasks (all stages). I aways load core source files and the ones we are working on up front. Around 20%, I make it autoprepare for a new session and clear.

Admittedly I have been doing this precautiously, based on anecdotal evidence, not because I had bad experiences with longer context deterioration myself.

In the brief time I had access to Fable 5, it went on long running tasks (>45 mins) into the 30-40% zone without apparent context coherence problems.

Re: Don't trust large context windows

#13

Post by hypfer »

mg wrote: post #9

Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an on…

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

Re: Don't trust large context windows

#14

Post by asd88 »

kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I’ve had similar experiences with Fable. 70%+ context used out of 1M, still sharp and no memory issues.

Re: Don't trust large context windows

#16

Post by kristianc »

nopurpose wrote: post #11
kristianc wrote: post #6

I'm getting a lot of mileage out of basically acting like the AI's Product Manager, and insisting that it writes up short PRDs for every feature we propose to build. That gives it a reference over time of everything that has been built, but also makes it less liable to drift with each one. Each one gets its own conversation. For me this is a happy medium between stopping it going off the rails but also making sure it…

Is it adhoc or you use more structured approaches like openspec? I also tend to work on a plan first, but it stays as in-session todo, which is hard to reference later.

It's ad hoc / my own framework, just found something which works for me. The exact structure is

- Work Mode - HITL/AFK

- Problem Statement

- Who It Affects - Primary / Secondary User

- User Stories

- Business Case

- Why Now

- Success Critera

- In Scope/Out of Scope [Out of Scope v. important)

- Thinnest Slice (This I've found super valuable, means you max out the amount of 'product' for your buck and avoid diminishing marginal returns or overbuilding. Often I will build this)

- Eigenfeature - What is the larger feature we _could_ (but probably won't) which would solve for this use case and other stuff I might not have thought of

- Technical Notes

- Deps

- Schema Changes

- Risks

- Final Recommendation [go / no go, including on scope]

There's a note in my Claude / Agents MD which says no net new feature gets introduced without this and I get it to move through a pipeline of folders (active, approved, shipped, proposed etc). All runs in a system of MD files and have even created a little MD Kanban from the metadata!

Re: Don't trust large context windows

#18

Post by fullstackchris »

kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

Thats another problem of this post, the author mentions Claude but not explicitely what models...

100k tokens "by lunch" is also not my finding, the newer models will hit that already right in the initial exploratory phase

Re: Don't trust large context windows

#19

Post by arcanemachiner »

kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

Opus 4.6 was on drugs past 200k, I skipped 4.7, 4.8 did good up to ~350k, and Fable did great beyond 400k, in my limited testing. The quality does appear to be trending upwards.

Re: Don't trust large context windows

#20

Post by arcanemachiner »

fullstackchris wrote: post #18
kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

Thats another problem of this post, the author mentions Claude but not explicitely what models... 100k tokens "by lunch" is also not my finding, the newer models will hit that already right in the initial exploratory phase

Really depends on the project.
Post reply on HN