Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't trust large context windows

garrit.xyz

Post reply on HN
Print view View original

41–50 of 211 posts

Re: Don't trust large context windows

#41

Post by loehnsberg »

hypfer wrote: post #13
mg wrote: post #9

Considerations about what goes on in agents internally will probably not be part of software development for long. Personally, I already see LLMs and agents as blackboxes. I give each feature request to multiple LLMs and then compare the results. I don't manually use "sessions" at all. I just look at the outcome. When I dislike it, I "git reset --hard", change my prompts and restart the feature request. To have an on…

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

Unless we do our own benchmarks, we have to take all the marketing fluff from the frontier labs at face value, and all public benchmarks degrade eventually as labs optimize towards them. OP’s approach is wasteful because it is brute force, but post says that an ELO is kept, so this is also an experiment, and I don‘t see what‘s wrong with that. You learn which model performs well in which settings which may save resources later. It‘s also wasteful to keep working with the wrong model/harness/tools for too long.

Re: Don't trust large context windows

#42

Post by jgilias »

mg wrote: post #31
hypfer wrote: post #13

Earlier quoted context omitted.

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

It is the other way round. In an interactive session, adding "Fine, but make the button red" after the model generated a first solution more than doubles the tokens used. As the model now not only gets the original code and the feature request but also the updated code plus the change request as input tokens. Sending a feature request to an LLM and then sending the feature request again with "The button shall be red"…

The cost is far from linear though. Because of prompt caching and the fact that generally output tokens are a lot more expensive than input tokens.

Re: Don't trust large context windows

#43

Post by Chirono »

mg wrote: post #31
hypfer wrote: post #13

Earlier quoted context omitted.

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

It is the other way round. In an interactive session, adding "Fine, but make the button red" after the model generated a first solution more than doubles the tokens used. As the model now not only gets the original code and the feature request but also the updated code plus the change request as input tokens. Sending a feature request to an LLM and then sending the feature request again with "The button shall be red"…

That’s usually not true due to caching. It may be true if you leave a large gap in between, but if you send “make it red” right after, then it’s purely incremental

Re: Don't trust large context windows

#44

Post by ryan_glass »

mg wrote: post #31
hypfer wrote: post #13

Earlier quoted context omitted.

This is an absolutely crazy wasteful thing to do considering the actual cost of all that inference and nothing to be proud of.

It is the other way round. In an interactive session, adding "Fine, but make the button red" after the model generated a first solution more than doubles the tokens used. As the model now not only gets the original code and the feature request but also the updated code plus the change request as input tokens. Sending a feature request to an LLM and then sending the feature request again with "The button shall be red"…

"Make the button red" probably doesn't need an LLM at all.

Re: Don't trust large context windows

#45

Post by [deleted] »

da_grift_shift wrote: post #37
kristianc wrote: post #16

Earlier quoted context omitted.

It's ad hoc / my own framework, just found something which works for me. The exact structure is - Work Mode - HITL/AFK - Problem Statement - Who It Affects - Primary / Secondary User - User Stories - Business Case - Why Now - Success Critera - In Scope/Out of Scope [Out of Scope v. important) - Thinnest Slice (This I've found super valuable, means you max out the amount of 'product' for your buck and avoid diminishin…

Is there back-and-forth? How long do these get? Can you share an example?

[deleted]

Re: Don't trust large context windows

#47

Post by SwellJoe »

Opus in recent versions is fine beyond 100k, but I usually do try to keep it under 200k.

But, this is also why so-called "memory" systems are usually a mistake that make the models dumber. They don't have memory, they only have context, and every irrelevant fact you shove into the context is less context for the problem. Less distractions, better results.

The way to have the agent remember things is to have it document its work, like a human developer would do if they wanted their project to be friendly to other developers working on it. Good developer docs with an index page and a good plan with checklists, in concise Markdown files, checked in to the repo is the ideal memory for models and the ideal docs you need to figure out WTF the model has been up to. Helps with code review, too, whether by humans or another model. There's no down side.

Re: Don't trust large context windows

#48

Post by RugnirViking »

da_grift_shift wrote: post #34
eterm wrote: post #25

Earlier quoted context omitted.

60k is tiny, if it's making recall mistakes that early then you might have some false memories or incorrect instructions in your CLAUDE.md. 60k isn't much bigger than the system prompt.

>you might have some false memories or incorrect instructions in your CLAUDE.md "YOU'RE HOLDING IT WRONG!"

did you internalize what was wrong with that quote when it was said? does it apply here?

Re: Don't trust large context windows

#50

Post by CjHuber »

Bolwin wrote: post #21
kelnos wrote: post #8

This has not been my experience with Opus since Anthropic released the 1M token context window for use under the subscription plans. I routinely push past 500k tokens, even sometimes up to around 800k tokens, and don't see this problem. I've seen it to some extent when getting truly near the limit, up around and above 900k tokens, though what I see isn't as severe as the author seems to see. (And I rarely fill the co…

I see this said often and find it insane given how many times I find opus models making basic recall mistakes at Personally I consider < 60k to be the smart zone for opus. This is worse for opus 4.7 and 4.8 cause of the more granular tokenizer

I'm always a bit confused when people say things like this. 60k token is often more than the initial context I feed the model with. And I don't think I ever had a productive session that began under 150k tokens.
Post reply on HN