Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context is the bottleneck for coding agents now

runnercode.com

Post reply on HN
Print view View original

121–130 of 193 posts

Re: Context is the bottleneck for coding agents now

#121

Post by sethhochberg »

davedx wrote: post #71
dingnuts wrote: post #55

Earlier quoted context omitted.

that's because a next token predictor can't "forget" context. That's just not how it works. You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want That's why they have a tendency to go ahead and do things you tell them not to do.. also IDK about yo…

Yeah I start a new session to mitigate this. Don’t keep hammering away - close the current chat/session whatever and restate the problem carefully in a new one.

I've had great luck with asking the current session to "summarize our goals, conversation, and other relevant details like git commits to this point in a compact but technically precise way that lets a new LLM pick up where we're leaving off".

The new session throws away whatever behind-the-scenes context was causing problems, but the prepared prompt gets the new session up and running more quickly especially if picking up in the middle of a piece of work that's already in progress.

Re: Context is the bottleneck for coding agents now

#122

Post by raincole »

aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

Humans have a very strong tendency (and have made tremendous collective efforts) to compress context. I'm not a neuroscientist but I believe it's called "chunk."

Language itself is a highly compressed form of compressed context. Like when you read "hoist with one's own petard" you don't just think about literal petard but the context behind this phrase.

Re: Context is the bottleneck for coding agents now

#123

Post by musebox35 »

Context is also a bottleneck in many human to human interactions as well so this is not surprising. Especially juniors often start by talking about their problems without providing adequate context about what they’re trying to accomplish or why they’re doing it.

Mind you, I was exactly like that when I started my career and it took quite a while and being on both sides of the conversation to improve. One difference is that it is not so easy to put oneself in the shoes of an LLM. Maybe I will improve with time. So far assuming the LLM is knowledgeable but not very smart has been the most effective strategy for my LLM interactions.

Re: Context is the bottleneck for coding agents now

#124

Post by SparkyMcUnicorn »

ljm wrote: post #65
SparkyMcUnicorn wrote: post #34

Earlier quoted context omitted.

I'll stop you right there. I've been using Claude Code for almost a year on production software with pretty large codebases. Both multi-repo and monorepo. Claude is able to create entire PRs for me that are clean, well written, and maintainable. Can it fail spectacularly? Yes, and it does sometimes. Can it be given good instructions and produce results that feel like magic? Also yes.

For finicky issues like that I often find that, in the time it takes to create a prompt with the necessary context, I was able to just make the one line tweak myself. In a way that is still helpful, especially if the act of putting the prompt together brought you to the solution organically. Beyond that, 'clean', 'well written' and 'maintainable' are all relative terms here. In a low quality, mega legacy codebase, th…

> For finicky issues like that I often find that, in the time it takes to create a prompt with the necessary context, I was able to just make the one line tweak myself.

I don't run into this problem. Maybe the type of code we're working on is just very different. In my experience, if a one-line tweak is the answer and I'm spending a lot of time tweaking a prompt, then I might be holding the tool wrong.

Agree on those terms being relative. Maybe a better way of putting it is that I'm very comfortable putting my name on it, deploying to production, and taking responsibility for any bugs.

Re: Context is the bottleneck for coding agents now

#125

Post by whstl »

lomase wrote: post #5
bhu8 wrote: post #3

IMHO, jumping from Level 2 to Level 5 is a matter of: - Better structured codebases - we need hierarchical codebases with minimal depth, maximal orthogonality and reasonable width. Think microservices. - Better documentation - most code documentations are not built to handle updates. We need a proper graph structure with few sources of truth that get propagated downstream. Again, some optimal sort of hierarchy is cru…

Can you show something you have built with that workflow?

I "vibe coded" a Gateway/Proxy server that did a lot of request enrichment and proprietary authz stuff that was previously in AWS services. The goal was to save money by having a couple high-performance servers instead of relying on cloud-native stuff.

I put "vibe coded" is in quotes because the code was heavily reviewed after the process, I helped when the agent got stuck (I know pedants will complain but ), and this was definitely not my first rodeo in this domain and I just wanted to see how far an agent could go.

In the end it had a few modifications and went into prod, but to be really fair it was actually fine!

One thing I vibe coded 100% and barely looked at the code until the end was a MacOS menubar app that shows some company stats. I wanted it in Swift but WITHOUT Xcode. It was super helpful in that regard.

Re: Context is the bottleneck for coding agents now

#126

Post by PantaloonFlames »

cma wrote: post #109
bgirard wrote: post #31

Earlier quoted context omitted.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

Not that this shouldn't be fixed in the model, but you can jump to an earlier point in claude code and on web chat interfaces to get it out of the context, just sometimes you have other important stuff you don't want it to lose.

Likewise Gemini CLI. There’s a way to backup to a prior state in the dialogue.

Re: Context is the bottleneck for coding agents now

#127

Post by J_Shelby_J »

I’m working on a project that has now outgrown the context window of even gpt-5 pro. I use code2prompt and ChatGPT with pro will reject the prompt as too large.

I’ve been trying to use shorter variable names. Maybe I should move unit tests into their own file and ignore them? It’s not idiomatic in Rust though and breaks visibility rules for the modules.

What we really need is for the agent to assemble the required context for the problem space. I suspect this is what coding agents will do if they don’t already.

Re: Context is the bottleneck for coding agents now

#128

Post by PantaloonFlames »

dragonwriter wrote: post #69
dingnuts wrote: post #55

Earlier quoted context omitted.

that's because a next token predictor can't "forget" context. That's just not how it works. You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want That's why they have a tendency to go ahead and do things you tell them not to do.. also IDK about yo…

This is where the distinction between “an LLM” and “a user-facing system backed by an LLM” becomes important; the latter is often much more than a naive system for maintaining history and reprompting the LLM with added context from new user input, and could absolutely incorporate a step which (using the same LLM with different prompting or completely different tooling) edited the context before presenting it to the L…

At least a few of the current coding agents have mechanisms that do what you describe.

Re: Context is the bottleneck for coding agents now

#129

Post by PantaloonFlames »

cjbgkagh wrote: post #77
davedx wrote: post #71

Earlier quoted context omitted.

Yeah I start a new session to mitigate this. Don’t keep hammering away - close the current chat/session whatever and restate the problem carefully in a new one.

There should be a simple button that allows you refine the context. A fresh LLM could generate a new context from the input and outputs of the chat history, then another fresh LLM can start over with that context.

You are saying “fresh LLM” but really I think you’re referring to a curated context. The existing coding agents have mechanisms to do this. Saving context to a file. Editing the file. Clearing all context except for the file. It’s sort of clunky now but it will get better and slicker.

Re: Context is the bottleneck for coding agents now

#130

Post by PantaloonFlames »

raincole wrote: post #122
aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

Humans have a very strong tendency (and have made tremendous collective efforts) to compress context. I'm not a neuroscientist but I believe it's called "chunk." Language itself is a highly compressed form of compressed context. Like when you read "hoist with one's own petard" you don't just think about literal petard but the context behind this phrase.

We don’t think of petards because no one knows what that is. :)
Post reply on HN