I don't think intelligence is increasing. Arbitrary benchmarks don't reflect real world usage. Even with all the context it could possibly have, these models still miss/hallucinate things. Doesn't make them useless, but saying context is the bottleneck is incorrect.
Context is the bottleneck for coding agents now
41–50 of 193 posts
Re: Context is the bottleneck for coding agents now
#42There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.
No, I think context itself is still an issue. Coding agents choke on our big C++ code-base pretty spectacularly if asked to reference large files.
Yeah, I have the same issue too. Even for a file with several thousand lines, they will "forget" earlier parts of the file they're still working in resulting in mistakes. They don't need full awareness of the context, but they need a summary of it so that they can go back and review relevant sections.
I have multiple things I'd love LLMs to attempt to do, but the context window is stopping me.
Re: Context is the bottleneck for coding agents now
#43There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.
This is a great insight. Any thoughts on how to address this problem?
For me? It's simple. Completely empty the context and rebuild focused on the new task at hand. It's painful, but very effective.
Re: Context is the bottleneck for coding agents now
#44Earlier quoted context omitted.
They have no ability to even perceive time, unless the system gives them timestamps for the current interaction and past interactions.
Which seems like a trivial addition if it's not there?
It is, but now you're burning a bit of context on something that might not be necessary, and potentially having the agent focus on time when it's not relevant. Not necessarily a bad idea, but as always, tradeoffs.
Re: Context is the bottleneck for coding agents now
#45IME speed is the biggest bottleneck. They simply can't navigate the code base fast enough.
grok-code-fast-1 is quite nice for this actually, its fast and cheap enough that you dont feel bad throwing entire threads away and trying again.
Re: Context is the bottleneck for coding agents now
#46Earlier quoted context omitted.
> Think microservices. Microservices should already be a last resort when you’ve either: a) hit technical scale that necessitates it b) hit organizational complexity that necessitates it Opting to introduce them sooner will almost certainly increase the complexity of your codebase prematurely (already a hallmark of LLM development). > Better documentation If this means reasoning as to why decisions are made then yes.…
> Opting to introduce them sooner will almost certainly increase the complexity of your codebase prematurely Agreed, but how else are you going to scale mostly AI written code? Relying mostly on AI agents gives you that organizational complexity. > Given how long gpt codex 5 has been out, there’s no way you’ve followed these practices for a reasonable enough time to consider them definitive Yeah, fair. Codex has been…
I understand why you made it microservices, people make that too even when not using LLMs, because it looks like it is more organized.
But in my experience a microservide architecture is orders of magnitud more complex to build and understand that a monolith.
If you, with the help of an LLM, strugle to keep a monolith organized, I am positive you will find even harder to build microservices.
Good luck in your journey, I hope you learn a ton!
Re: Context is the bottleneck for coding agents now
#47Earlier quoted context omitted.
This is a great insight. Any thoughts on how to address this problem?
It has to be addressed architecturally with some sort of extension to transformers that can focus the attention on just the relevant context. People have tried to expand context windows by reducing the O(n^2) attention mechanism to something more sparse and it tends to perform very poorly. It will take a fundamental architectural change.
Can one instruct an LLM to pick the parts of the context that will be relevant going forward? And then discard the existing context, replacing it with the new 'summary'?
Re: Context is the bottleneck for coding agents now
#48> Level 2 - One commit - Cursor and Claude Code work well for tasks in this size range. I'll stop ya right there. Spending the past few weeks fixing bugs in a big multi-tier app (which is what any production software is this days). My output per bug is always one commit, often one line. Claude is an occasional help, nothing more. Certainly not generating the commit for me!
This is interesting, and I'd say you're not the target audience. If you want the code Claude writes to be line-by-line what you think is most appropriate as a human, you're not going to get it.
You have to be willing to accept "close-ish and good enough" to what you'd write yourself. I would say that most of the time I spend with Claude is to get from its initial try to "close-ish and good enough". If I was working on tiny changes of just a few lines, it would definitely be faster just to write them myself. It's the hundreds of lines of boilerplate, logging, error handling, etc. that makes the trade-off close to worth it.
Re: Context is the bottleneck for coding agents now
#49> Intelligence is rapidly improving with each model release. Are we still calling it intelligence?
Just a reminder that language is flexible.
Re: Context is the bottleneck for coding agents now
#50This has been the case for a while. Attempting to code API connections via Vibe-Coding will leave you pulling your hair out if you don't take the time to scrape all relevant documentation and include said documentation in the prompt. This is the case whether it's major APIs like Shopify, or more niche ones like warehousing software (Cin7 or something similar).
The context pipeline is a major problem in other fields as well, not just programming. In healthcare, the next billion-dollar startup will likely be the one that cracks the personal health pipeline, enabling people to chat with GPT-6 PRO while seamlessly bringing their entire lifetime of health context into every conversation.