Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust large context windows

garrit.xyz

Post reply on HN
Print view View original

141–150 of 211 posts

Re: Don't trust large context windows

#141

Post by kuboble »

diab0lic wrote: post #102
kuboble wrote: post #96

Earlier quoted context omitted.

No it's not. It seems that people have different workflows or repos, or memories or prompts or expectations.

For what it’s worth, as a third party I read your and qsera’s comments as saying the same thing.

Maybe I misread the comment then.

I read it as a models performance being random and observed differences in the opinions are the results of the overinterpretation of the random outcomes.

I think however that some people seem to be always lucky which indicates that it is not random but rather some fixed differences between people and their environments.

Re: Don't trust large context windows

#142

Post by Bolwin »

eterm wrote: post #25
Bolwin wrote: post #21

Earlier quoted context omitted.

I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer

60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md. 60k isn't much bigger than the system prompt.

I don't use Claude Code. I use my own handwritten agent (formerly using Pi) and know every token that goes into it. There are zero memories to confuse it. The system prompt is 200 tokens and completely self consistent.

Plus I've found that the only time models go above 100k tokens anyway is when they've started looping at which point it's much better to go back anyway.

Anecdotally most models know their recall is terrible (or have been trained to act as such), that's why they constantly reread files before editing or while reasoning.

Re: Don't trust large context windows

#143

Post by JCTheDenthog »

dindunuf wrote: post #140
dofm wrote: post #123

I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…

first of all, LLM-assisted coding is less than 3 years old. 3 years ago all we had was GPT-4 with 8192 token context, which wasn't enough for most things. and second of all... >Any shared sense of rigour is just completely torpedoed by the LLM world, particularly the cloud LLM world it seems, and we are reduced to cargo culting. Nobody is any more right or wrong than anyone else. what "sense of rigour"? it's way too…

>what "sense of rigour"? it's way too soon to put those rose-tinted glasses on.

I don't think OP is claiming that prior to LLM coding everything in the software development world was super rigorous (I assume that's effectively what you mean with the "rose-tinted glasses" comment). But rigor was actually possible and in a deterministic way too, which is fundamentally impossible with LLMs. You can build all kinds of guardrails and processes around LLMs that make it somewhat approach rigor again, but it's still fundamentally based on a bunch of statistical probabilities instead of deterministic, repeatable results.

All of the methods I see to mitigate the fundamental and inherent issues of LLMs seem roughly equivalent to the kind of crap you see in astrology groups or palm reading etc. You need Venus and Mercury to be in alignment while Mars is retrograde if you want to be able to get the right results from your token predictor.

Re: Don't trust large context windows

#144

Post by esperent »

CommieBobDole wrote: post #128
dofm wrote: post #123

I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…

This has always been a thing with IT advice, though - the more complex a system and the outcome, the harder it is to clearly define "better" or "worse". Add in the fact that LLMs are intensely and emphatically non-deterministic and LLM guidance basically becomes gardening advice. Heck, even the 'benchmarks' are mostly somebody's attempt to crystallize their vibes with varying amounts of success.

> LLMs are intensely and emphatically non-deterministic and LLM guidance basically becomes gardening advice

Have you ever tried doing evals on moderately complex but bounded tasks?

I spent some time doing it when testing these "token reducing" tools like Headroom, RTK etc. as well as customizing my Pi tools. What I found interesting was that despite LLMs being deterministic, for a given toolset and prompt, the results were highly consistent for a given eval, across multiple models (I tested at the time using GPT 5.4 mini, 5.5, 5.3 codex, Gemini 3 flash, initially running sets of 5 evals on each task but once I realized how consistent the results were, dropping to sets of 3.

Aside: in my tests, RTK and Headroom made the overall context use higher for roughly equivalent results. The context use for those specific toolcalls went down but the number of model turns and overall context use went up.

Re: Don't trust large context windows

#145

Post by dofm »

JCTheDenthog wrote: post #143
dindunuf wrote: post #140

Earlier quoted context omitted.

first of all, LLM-assisted coding is less than 3 years old. 3 years ago all we had was GPT-4 with 8192 token context, which wasn't enough for most things. and second of all... >Any shared sense of rigour is just completely torpedoed by the LLM world, particularly the cloud LLM world it seems, and we are reduced to cargo culting. Nobody is any more right or wrong than anyone else. what "sense of rigour"? it's way too…

>what "sense of rigour"? it's way too soon to put those rose-tinted glasses on. I don't think OP is claiming that prior to LLM coding everything in the software development world was super rigorous (I assume that's effectively what you mean with the "rose-tinted glasses" comment). But rigor was actually possible and in a deterministic way too, which is fundamentally impossible with LLMs. You can build all kinds of gu…

Astrology? And I thought I was being overly harsh with the 3D printing comparison ;-)

Re: Don't trust large context windows

#146

Post by Bolwin »

cubefox wrote: post #124

The problem with "context rot" is that its existence and severity is purely anecdotal. As far as I know, nobody has actually measured context rot systematically. The only thing we know is that memory degrades somewhat in long contexts, via things like needle in haystack tests. But that's not the same issue. Context rot is usually taken to mean that the model gets dumber even if it doesn't need to remember specific th…

It's pretty hard to measure because most context rot comes from related context and the model has to be able to figure which parts are truly relevant, which ones are relevant but stale, which ones to ignore etc.

Each relevant thing is basically a rule. Trying to so something with 500 rules is what's hard.

If you take a standard benchmark and just prepend a random book to it, it will not capture that

Re: Don't trust large context windows

#147

Post by ninjalanternshk »

JCTheDenthog wrote: post #143
dindunuf wrote: post #140

Earlier quoted context omitted.

first of all, LLM-assisted coding is less than 3 years old. 3 years ago all we had was GPT-4 with 8192 token context, which wasn't enough for most things. and second of all... >Any shared sense of rigour is just completely torpedoed by the LLM world, particularly the cloud LLM world it seems, and we are reduced to cargo culting. Nobody is any more right or wrong than anyone else. what "sense of rigour"? it's way too…

>what "sense of rigour"? it's way too soon to put those rose-tinted glasses on. I don't think OP is claiming that prior to LLM coding everything in the software development world was super rigorous (I assume that's effectively what you mean with the "rose-tinted glasses" comment). But rigor was actually possible and in a deterministic way too, which is fundamentally impossible with LLMs. You can build all kinds of gu…

Aren’t human coders non-deterministic? There’s no guarantee two people with otherwise identical levels of experience will always write identical code.

Any software engineering practice that had enough review and feedback to work with humans should work more or less the same with AI coders.

It’s when someone tries replacing an entire team or an entire process with a single prompt that they get in trouble.

Re: Don't trust large context windows

#149

Post by 01100011 »

dofm wrote: post #123

I guess I am mostly enjoying learning the fundamentals of AI stuff, even though I disagree with the direction it is going. But I am struggling to put into words how alarming I find the comments on threads like this — all sorts of good-natured anecdotes about how XYZ works for them that are more like the suggestions in pet care or cookery threads on Facebook. (Or worse still, like any Facebook 3D printing group: anyon…

The arbitrary and non-deterministic nature of LLM workflows gives me full on ick. As an old embedded/systems guy I have always prioritized determinism and repeatability in my workflows.

But damn, agents are amazing and I'm enjoying being a "thought process designer". I'm not going back. Even if AI development stops today my career will never be the same.

Re: Don't trust large context windows

#150

Post by pdantix »

kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

agreed. the claudes have been getting better and better with every release in this regard.

opus 4.5 would start failing tool calls when approaching its 200k limit, opus 4.6 could get to ~300k before getting confused, opus 4.7 i could stretch to around 400k the dumb zone started, with opus 4.8 i've had sessions get over 500k comfortably.

admittedly we only had limited time with fable, but i had a couple sessions get into 800-900k just fine.

Post reply on HN