Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context is the bottleneck for coding agents now

runnercode.com

Post reply on HN
Print view View original

31–40 of 193 posts

Re: Context is the bottleneck for coding agents now

#31

Post by bgirard »

aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

Re: Context is the bottleneck for coding agents now

#32

Post by ninetyninenine »

lomase wrote: post #25
ninetyninenine wrote: post #17

Earlier quoted context omitted.

I addressed this. The AI needs to examine every code change going in whether that code change comes from AI or not and edit the summaries accordingly. This is something humans dont actually do. We aren’t aware of every change and we don’t have updated documentation of every change so the LLM will be doing better in this regard.

I mean... have you ever heard of this small tool called GIT that people use to track code changes?

I’m not talking about git diffs. I’m talking about the summaries of context. Every commit the ai needs to update the summaries and notes it took about the code.

Did you read the entirety of what I wrote? Please read.

Say the AI left a 5 line summary of a 300 line piece of code. You as a human update that code. What I am saying specifically is this: when you do the change, The AI then sees this and updates the summary. So AI needs to be interacting with every code change whether or not you used it to vibe code.

The next time the AI needs to know what this function does, it doesn’t need to read the entire 300 line function. It reads the 5 line summary, puts it in the context window and moves on with chain of thought. Understand?

This is what shrinks the context. Humans don’t have unlimited context either. We have vague fuzzy memories of aspects of the code and these “notes” effectively make coding agents do the same thing.

Re: Context is the bottleneck for coding agents now

#33

Post by multiplegeorges »

wat10000 wrote: post #13
maerF0x0 wrote: post #7

I've noticed that chatgpt doesnt seem to be very good at understanding elapsed time. I have some long running threads and unless i prompt it with elapsed time ("it's now 7 days later") the responses act like it was 1 second after the last message. I think this might be a good leap for agents, the ability to not just review a doc in it's current state, but to keep in context/understanding the full evolution of a docum…

They have no ability to even perceive time, unless the system gives them timestamps for the current interaction and past interactions.

Which seems like a trivial addition if it's not there?

Re: Context is the bottleneck for coding agents now

#34

Post by SparkyMcUnicorn »

marstall wrote: post #9

> Level 2 - One commit - Cursor and Claude Code work well for tasks in this size range. I'll stop ya right there. Spending the past few weeks fixing bugs in a big multi-tier app (which is what any production software is this days). My output per bug is always one commit, often one line. Claude is an occasional help, nothing more. Certainly not generating the commit for me!

I'll stop you right there. I've been using Claude Code for almost a year on production software with pretty large codebases. Both multi-repo and monorepo.

Claude is able to create entire PRs for me that are clean, well written, and maintainable.

Can it fail spectacularly? Yes, and it does sometimes. Can it be given good instructions and produce results that feel like magic? Also yes.

Re: Context is the bottleneck for coding agents now

#35

Post by perplex »

bhu8 wrote: post #3

IMHO, jumping from Level 2 to Level 5 is a matter of: - Better structured codebases - we need hierarchical codebases with minimal depth, maximal orthogonality and reasonable width. Think microservices. - Better documentation - most code documentations are not built to handle updates. We need a proper graph structure with few sources of truth that get propagated downstream. Again, some optimal sort of hierarchy is cru…

I've been using claude on two codebases, one with good layering and clean examples, the other not so much. I get better output from the LLM with good context and clean examples and documentation. Not surprising that clarity in code benefits both humans and machines.

Re: Context is the bottleneck for coding agents now

#36

Post by _joel »

I'm making a pretty complex project using claude. I tried claude flow and some other orchestrators but they produced garbage. Have found using github issues to track the progress as comments works fairly well, the PR's can get large comment wise (especially if you have gemini code assist, recommeded as another code review judge), so be mindful of that (that will blow the context window). Using a fairly lean CLAUDE.md and a few mcps (context7 and consult7 with gemini for longer lookups). works well too. Although be prepared to tell it to reread CLAUDE.md a few conversations deep as it loses it. It's working fairly well so far, it feels a bit akin to herding cats sometimes and be prepared to actually read the code it's making, or the important bits at least.

Re: Context is the bottleneck for coding agents now

#37

Post by multiplegeorges »

wat10000 wrote: post #20
ninetyninenine wrote: post #6

Context is a bottleneck for humans as well. We don’t have full context when going through the code because we can’t hold full context. We summarize context and remember summarizations of it. Maybe we need to do this with the LLM. Chain of thought sort of does this but it’s not deliberate. The system prompt needs to mark this as a deliberate task of building summaries and notes notes of the entire code base and this s…

They need a proper memory. Imagine you're a very smart, skilled programmer but your memory resets every hour. You could probably get something done by making extensive notes as you go along, but you'll still be smoked by someone who can actually remember what they were doing in the morning. That's the situation these coding agents are in. The fact that they do as well as they do is remarkable, considering.

Basically, LLMs are the guy from Memento.

Re: Context is the bottleneck for coding agents now

#38

Post by pu_pe »

neutronicus wrote: post #12
CardenB wrote: post #4

Earlier quoted context omitted.

I know it isn’t your question exactly, and you probably know this, but the models for coding assist tools are generally fine tunes of models for coding specific purposes. Example: in OpenAI codex they use GPT-5-codex

I think the question is, can I throw a couple thousand bucks of GPU time at fine-tuning a model to have knowledge of our couple million lines of C++ baked into the weights instead of needing to fuck around with "Context Engineering". Like, how feasible is it for a mid-size corporation to use a technique like LoRA, mentioned by GP, to "teach" (say, for example) Kimi K2 about a large C++ codebase so that individual eng…

I'm curious about it too. I think there are two bottlenecks, one is that training a relatively large LLM can be resource-intensive (so people go for RAGs and other shortcuts), and making it finetuned to your use cases might make it dumber overall.

Re: Context is the bottleneck for coding agents now

#39

Post by bhu8 »

skaosobab wrote: post #26
bhu8 wrote: post #3

IMHO, jumping from Level 2 to Level 5 is a matter of: - Better structured codebases - we need hierarchical codebases with minimal depth, maximal orthogonality and reasonable width. Think microservices. - Better documentation - most code documentations are not built to handle updates. We need a proper graph structure with few sources of truth that get propagated downstream. Again, some optimal sort of hierarchy is cru…

> Think microservices. Microservices should already be a last resort when you’ve either: a) hit technical scale that necessitates it b) hit organizational complexity that necessitates it Opting to introduce them sooner will almost certainly increase the complexity of your codebase prematurely (already a hallmark of LLM development). > Better documentation If this means reasoning as to why decisions are made then yes.…

> Opting to introduce them sooner will almost certainly increase the complexity of your codebase prematurely

Agreed, but how else are you going to scale mostly AI written code? Relying mostly on AI agents gives you that organizational complexity.

> Given how long gpt codex 5 has been out, there’s no way you’ve followed these practices for a reasonable enough time to consider them definitive

Yeah, fair. Codex has been out for less than 2 weeks at this point. I was relying on gpt-5 in August and opus before that.

Re: Context is the bottleneck for coding agents now

#40

Post by throwup238 »

ray__ wrote: post #18
aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

This is a great insight. Any thoughts on how to address this problem?

It has to be addressed architecturally with some sort of extension to transformers that can focus the attention on just the relevant context.

People have tried to expand context windows by reducing the O(n^2) attention mechanism to something more sparse and it tends to perform very poorly. It will take a fundamental architectural change.

Post reply on HN