Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context is the bottleneck for coding agents now

runnercode.com

Post reply on HN
Print view View original

61–70 of 193 posts

Re: Context is the bottleneck for coding agents now

#61

Post by RedNifre »

lomase wrote: post #5
bhu8 wrote: post #3

IMHO, jumping from Level 2 to Level 5 is a matter of: - Better structured codebases - we need hierarchical codebases with minimal depth, maximal orthogonality and reasonable width. Think microservices. - Better documentation - most code documentations are not built to handle updates. We need a proper graph structure with few sources of truth that get propagated downstream. Again, some optimal sort of hierarchy is cru…

Can you show something you have built with that workflow?

I vibe coded an invoice generator by first vibe coding a "template" command line tool as a bash script that substitutes {{words}} in a libre office writer document (those are just zipped xml files, so you can unpack them to a temp directory and substitute raw text without xml awareness), and in the end it calls libre office's cli to convert it to pdf. I also asked the AI to generate a documentation text file, so that the next AI conversation could use the command as a black box.

The vibe coded main invoice generator script then does the calendar calculations to figure out the pay cycle and examines existing invoices in the invoice directory to determine the next invoice number (the invoice number is in the file name, so it doesn't need to open the files). When it is done with the calculations, it uses the template command to generate the final invoice.

This is a very small example, but I do think that clearly defined modules/microservices/libraries are a good way to only put the relevant work context into the limited context window.

It also happens to be more human-friendly, I think?

Re: Context is the bottleneck for coding agents now

#62

Post by revel »

This is one cause but another is that agents are mostly trained using the same sets of problems. There are only so many open source projects that can be used for training (ie. benchmarks). There's huge oversampling for a subset of projects like pandas and nothing at all for proprietary datasets. This is a huge problem!

If you want your agent to be really good at working with dates in a functional way or know how to deal with the metric system (as examples), then you need to train on those problems, probably using RFT. The other challenge is that even if you have this problem set in testable fashion running at scale is hard. Some benchmarks have 20k+ test cases and can take well over an hour to run. If you ran each test case sequentially it would take over 2 years to complete.

Right now the only company I'm aware of that lets you do that at scale is runloop (disclaimer, I work there).

Re: Context is the bottleneck for coding agents now

#63

Post by atonse »

ray__ wrote: post #18
aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

This is a great insight. Any thoughts on how to address this problem?

Do we know if LLMs understand the concept of time? (like i told you this in the past, but what i told you later should supersede it?)

I know there classes of problems that LLMs can't natively handle (like doing math, even simple addition... or spatial reasoning, I would assume time's in there too). There are ways they can hack around this, like writing code that performs the math.

But how would you do that for chronological reasoning? Because that would help with compacting context to know what to remember and what not.

Re: Context is the bottleneck for coding agents now

#64

Post by bhu8 »

lomase wrote: post #46
bhu8 wrote: post #39

Earlier quoted context omitted.

> Opting to introduce them sooner will almost certainly increase the complexity of your codebase prematurely Agreed, but how else are you going to scale mostly AI written code? Relying mostly on AI agents gives you that organizational complexity. > Given how long gpt codex 5 has been out, there’s no way you’ve followed these practices for a reasonable enough time to consider them definitive Yeah, fair. Codex has been…

I understand why you made it microservices, people make that too even when not using LLMs, because it looks like it is more organized. But in my experience a microservide architecture is orders of magnitud more complex to build and understand that a monolith. If you, with the help of an LLM, strugle to keep a monolith organized, I am positive you will find even harder to build microservices. Good luck in your journey…

Noted. Thanks!

Re: Context is the bottleneck for coding agents now

#65

Post by ljm »

SparkyMcUnicorn wrote: post #34
marstall wrote: post #9

> Level 2 - One commit - Cursor and Claude Code work well for tasks in this size range. I'll stop ya right there. Spending the past few weeks fixing bugs in a big multi-tier app (which is what any production software is this days). My output per bug is always one commit, often one line. Claude is an occasional help, nothing more. Certainly not generating the commit for me!

I'll stop you right there. I've been using Claude Code for almost a year on production software with pretty large codebases. Both multi-repo and monorepo. Claude is able to create entire PRs for me that are clean, well written, and maintainable. Can it fail spectacularly? Yes, and it does sometimes. Can it be given good instructions and produce results that feel like magic? Also yes.

For finicky issues like that I often find that, in the time it takes to create a prompt with the necessary context, I was able to just make the one line tweak myself.

In a way that is still helpful, especially if the act of putting the prompt together brought you to the solution organically.

Beyond that, 'clean', 'well written' and 'maintainable' are all relative terms here. In a low quality, mega legacy codebase, the results are gonna be dogshit without an intense amount of steering.

Re: Context is the bottleneck for coding agents now

#66

Post by magicalhippo »

throwup238 wrote: post #40
ray__ wrote: post #18

Earlier quoted context omitted.

This is a great insight. Any thoughts on how to address this problem?

It has to be addressed architecturally with some sort of extension to transformers that can focus the attention on just the relevant context. People have tried to expand context windows by reducing the O(n^2) attention mechanism to something more sparse and it tends to perform very poorly. It will take a fundamental architectural change.

I'm not an expert but it seemed fairly reasonable to me that a hierarchical model would be needed to approach what humans can do, as that's basically how we process data as well.

That is, humans usually don't store exactly what was written in as sentence five paragraphs ago, but rather the concept or idea conveyed. If we need details we go back and reread or similar.

And when we write or talk, we form first an overall thought about what to say, then we break it into pieces and order the pieces somewhat logically, before finally forming words that make up sentences for each piece.

From what I can see there's work on this, like this[1] and this[2] more recent paper. Again not an expert so can't comment on the quality of the references, just some I found.

[1]: https://aclanthology.org/2022.findings-naacl.117/

[2]: https://aclanthology.org/2025.naacl-long.410/

Re: Context is the bottleneck for coding agents now

#67

Post by cuttothechase »

It is pretty clear that the long horizon tasks are difficult for coding agents and that is a fundamental limitation of how probabilistic word generation works either with transformer or any other architecture. The errors propagate and multiply and becomes open ended.

However, the limitation can be masqueraded using layering techniques where output of one agent is fed as an input to another using consensus for verification or other techniques to the nth degree to minimize errors. But this is a bit like the story of a boy with a finger in the dike. Yes, you can spawn as many boys but there is a cost associated that would keep growing and wont narrow down.

It has nothing to do with contexts or window of focus or any other human centric metric. This is what the architecture is supposed to do and it does so perfectly.

Re: Context is the bottleneck for coding agents now

#68

Post by davedx »

> It needs to understand product and business requirements

Yeah this is the really big one - kind of buried the lede a little there :)

Understanding product and business requirements traditionally means communicating (either via docs and specs or directly with humans) with a bunch of people. One of the differences between a junior and senior is being able to read between the lines of a github or jira issue and know that more information needs to be teased out from… somewhere (most likely someone).

I’ve noticed that when working with AI lately I often explicitly tell them “if you need more information or context ask me before writing code”, or variations thereof. Because LLMs, like less experienced engineers, tend to think the only task is to start writing code immediately.

It will get solved though, there’s no magic in it, and LLMs are well equipped by design to communicate!

Re: Context is the bottleneck for coding agents now

#69

Post by dragonwriter »

dingnuts wrote: post #55
bgirard wrote: post #31

Earlier quoted context omitted.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

that's because a next token predictor can't "forget" context. That's just not how it works. You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want That's why they have a tendency to go ahead and do things you tell them not to do.. also IDK about yo…

This is where the distinction between “an LLM” and “a user-facing system backed by an LLM” becomes important; the latter is often much more than a naive system for maintaining history and reprompting the LLM with added context from new user input, and could absolutely incorporate a step which (using the same LLM with different prompting or completely different tooling) edited the context before presenting it to the LLM to generate the response to the user. And such a system could, by that mechanism, “forget” selected context in the process.

Re: Context is the bottleneck for coding agents now

#70

Post by suninsight »

aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

[flagged]
Post reply on HN