Context has been the bottleneck since the beginning
Context is the bottleneck for coding agents now
11–20 of 193 posts
Re: Context is the bottleneck for coding agents now
#12Has anyone tried making coding agent LoRas yet, project-specific and/or framework-specific?
I know it isn’t your question exactly, and you probably know this, but the models for coding assist tools are generally fine tunes of models for coding specific purposes. Example: in OpenAI codex they use GPT-5-codex
I think the question is, can I throw a couple thousand bucks of GPU time at fine-tuning a model to have knowledge of our couple million lines of C++ baked into the weights instead of needing to fuck around with "Context Engineering".
Like, how feasible is it for a mid-size corporation to use a technique like LoRA, mentioned by GP, to "teach" (say, for example) Kimi K2 about a large C++ codebase so that individual engineers don't need to learn the black art of "context engineering" and can just ask it questions.
Re: Context is the bottleneck for coding agents now
#13I've noticed that chatgpt doesnt seem to be very good at understanding elapsed time. I have some long running threads and unless i prompt it with elapsed time ("it's now 7 days later") the responses act like it was 1 second after the last message. I think this might be a good leap for agents, the ability to not just review a doc in it's current state, but to keep in context/understanding the full evolution of a docum…
They have no ability to even perceive time, unless the system gives them timestamps for the current interaction and past interactions.
Re: Context is the bottleneck for coding agents now
#14There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this.
Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.
Re: Context is the bottleneck for coding agents now
#15> Intelligence is rapidly improving with each model release.
Are we still calling it intelligence?
Re: Context is the bottleneck for coding agents now
#16IMHO, jumping from Level 2 to Level 5 is a matter of: - Better structured codebases - we need hierarchical codebases with minimal depth, maximal orthogonality and reasonable width. Think microservices. - Better documentation - most code documentations are not built to handle updates. We need a proper graph structure with few sources of truth that get propagated downstream. Again, some optimal sort of hierarchy is cru…
Can you show something you have built with that workflow?
Of course not.
Re: Context is the bottleneck for coding agents now
#17Context is a bottleneck for humans as well. We don’t have full context when going through the code because we can’t hold full context. We summarize context and remember summarizations of it. Maybe we need to do this with the LLM. Chain of thought sort of does this but it’s not deliberate. The system prompt needs to mark this as a deliberate task of building summaries and notes notes of the entire code base and this s…
> remember summarizations yes, and if you're an engineering manager you retain _out of date_ summarizations, often materially out of date.
I addressed this. The AI needs to examine every code change going in whether that code change comes from AI or not and edit the summaries accordingly.
This is something humans dont actually do. We aren’t aware of every change and we don’t have updated documentation of every change so the LLM will be doing better in this regard.
Re: Context is the bottleneck for coding agents now
#18There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.
This is a great insight. Any thoughts on how to address this problem?
Re: Context is the bottleneck for coding agents now
#19> Intelligence is rapidly improving with each model release. Are we still calling it intelligence?
I can feel the ground rumbling as thousands approach to engage in a "name the trait" style debate..
Re: Context is the bottleneck for coding agents now
#20Context is a bottleneck for humans as well. We don’t have full context when going through the code because we can’t hold full context. We summarize context and remember summarizations of it. Maybe we need to do this with the LLM. Chain of thought sort of does this but it’s not deliberate. The system prompt needs to mark this as a deliberate task of building summaries and notes notes of the entire code base and this s…
They need a proper memory. Imagine you're a very smart, skilled programmer but your memory resets every hour. You could probably get something done by making extensive notes as you go along, but you'll still be smoked by someone who can actually remember what they were doing in the morning. That's the situation these coding agents are in. The fact that they do as well as they do is remarkable, considering.