Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context is the bottleneck for coding agents now

runnercode.com

Post reply on HN
Print view View original

71–80 of 193 posts

Re: Context is the bottleneck for coding agents now

#71

Post by davedx »

dingnuts wrote: post #55
bgirard wrote: post #31

Earlier quoted context omitted.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

that's because a next token predictor can't "forget" context. That's just not how it works. You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want That's why they have a tendency to go ahead and do things you tell them not to do.. also IDK about yo…

Yeah I start a new session to mitigate this. Don’t keep hammering away - close the current chat/session whatever and restate the problem carefully in a new one.

Re: Context is the bottleneck for coding agents now

#72

Post by KeatonDunsford »

Here's a project I've been working on the past 2 weeks and only yesterday did I unify everything entirely while in Cursor Claude-4-Sonnet-1M MAX mode and I am pretty astounded with the results, Cursor usage dashboard tells me many of my prompts are 700k-1m context for around $0.60-$0.90 USD each, it adds up fast but wow it's extraordinary

https://github.com/foolsgoldtoshi-star/foolsgoldtoshi-star-p...

_ _ kae3g

Re: Context is the bottleneck for coding agents now

#73

Post by AnotherGoodName »

Someone1234 wrote: post #42
neutronicus wrote: post #24

Earlier quoted context omitted.

No, I think context itself is still an issue. Coding agents choke on our big C++ code-base pretty spectacularly if asked to reference large files.

Yeah, I have the same issue too. Even for a file with several thousand lines, they will "forget" earlier parts of the file they're still working in resulting in mistakes. They don't need full awareness of the context, but they need a summary of it so that they can go back and review relevant sections. I have multiple things I'd love LLMs to attempt to do, but the context window is stopping me.

I do take that as a sign to refactor when it happens though. Even if not for the sake of LLM compatibility with the codebase it cuts down merge conflicts to refactor large files.

In fact I've found LLMs are reasonable at the simple task of refactoring a large file into smaller components with documentation on what each portion does even if they can't get the full context immediately. Doing this then helps the LLM later. I'm also of the opinion we should be making codebases LLM compatible. So if it happens i direct the LLM that way for 10mins and then get back to the actual task once the codebase is in a more reasonable state.

Re: Context is the bottleneck for coding agents now

#74

Post by suninsight »

neutronicus wrote: post #24
aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

No, I think context itself is still an issue. Coding agents choke on our big C++ code-base pretty spectacularly if asked to reference large files.

[dead]

Re: Context is the bottleneck for coding agents now

#75

Post by atonse »

neutronicus wrote: post #24
aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

No, I think context itself is still an issue. Coding agents choke on our big C++ code-base pretty spectacularly if asked to reference large files.

I've found situations where a file was too big, and then it tries to grep for what might be useful in that file.

I could see in C++ it getting smarter about first checking the .h files or just grepping for function documentation, before actually trying to pull out parts of the file.

Re: Context is the bottleneck for coding agents now

#76

Post by KeatonDunsford »

wat10000 wrote: post #20
ninetyninenine wrote: post #6

Context is a bottleneck for humans as well. We don’t have full context when going through the code because we can’t hold full context. We summarize context and remember summarizations of it. Maybe we need to do this with the LLM. Chain of thought sort of does this but it’s not deliberate. The system prompt needs to mark this as a deliberate task of building summaries and notes notes of the entire code base and this s…

They need a proper memory. Imagine you're a very smart, skilled programmer but your memory resets every hour. You could probably get something done by making extensive notes as you go along, but you'll still be smoked by someone who can actually remember what they were doing in the morning. That's the situation these coding agents are in. The fact that they do as well as they do is remarkable, considering.

This is precisely how I go about my usage pattern with Cursor that I already have, I structure my repo declaratively with a Clojure and Nix build pipeline so when my context maxes out for a chat session, the repo is self-evident self-documented enough that a new chat session automatically has a heightened context

- - kae3g

Re: Context is the bottleneck for coding agents now

#77

Post by cjbgkagh »

davedx wrote: post #71
dingnuts wrote: post #55

Earlier quoted context omitted.

that's because a next token predictor can't "forget" context. That's just not how it works. You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want That's why they have a tendency to go ahead and do things you tell them not to do.. also IDK about yo…

Yeah I start a new session to mitigate this. Don’t keep hammering away - close the current chat/session whatever and restate the problem carefully in a new one.

There should be a simple button that allows you refine the context. A fresh LLM could generate a new context from the input and outputs of the chat history, then another fresh LLM can start over with that context.

Re: Context is the bottleneck for coding agents now

#79

Post by reclusive-sky »

asdev wrote: post #41

I don't think intelligence is increasing. Arbitrary benchmarks don't reflect real world usage. Even with all the context it could possibly have, these models still miss/hallucinate things. Doesn't make them useless, but saying context is the bottleneck is incorrect.

I agree, I often see Opus 4.1 and GPT5 (Thinking) make astoundingly stupid decisions with full confidence, even on trivial tasks requiring minimal context. Assuming they would make better decisions "if only they had more context" is a fallacy

Re: Context is the bottleneck for coding agents now

#80

Post by AnotherGoodName »

alastairr wrote: post #57

I agree, and I think intent behind the code is the most important part in missing context. You can sometimes infer intent from code, but usually code is a snapshot of an expression of an evolving intent.

I've started making sure my codebase is "LLM compatible". This means everything has documentation and the reasons for doing things a certain way and not another are documented in code. Funnily enough i do this documentation work with LLMs.

Eg. "Refactor this large file into meaningful smaller components where appropriate and add code documentation on what each small component is intended to achieve." The LLM can usually handle this well (with some oversight of course). I also have instructions to document each change and why in code in the LLMs instructions.md

If the LLM does create a regression i also ask the LLM to add code documentation in the code to avoid future regressions, "Important: do not do X here as it will break Y" which again seems to help since the LLM will see that next time right there in the portion of code where it's important.

None of this verbosity in the code itself is harmful to human readers either which is nice. The end result is the codebase becomes much easier for LLMs to work with.

I suspect LLM compatibility may be a metric we measure codebases in the future as we learn more and more how to work with them. Right now LLMs themselves often create very poor LLM compatible code but by adding some more documentation in the code itself they can do much better.

Post reply on HN