Notably, all of this information would be very helpful if written down as documentation in the first place. Maybe this will encourage people to do that?
Context is the bottleneck for coding agents now
181–190 of 193 posts
Re: Context is the bottleneck for coding agents now
#182Earlier quoted context omitted.
I agree, I often see Opus 4.1 and GPT5 (Thinking) make astoundingly stupid decisions with full confidence, even on trivial tasks requiring minimal context. Assuming they would make better decisions "if only they had more context" is a fallacy
Is there a good example you could provide of that? I just haven’t seen that personally so I’d be interested in any examples on these current models. I’m sure we all remember in the early days lots of examples of stupidity being posted and it was interesting. It be great if people kept doing that so we could get a better sense of which types of problems they are failing with astounding levels of stupidity on.
I've had every single LLM I tried (Opus, Sonnet, GPT-5-(codex) and Grok light) all tell me that Go embeds[0] support relative paths UPWARDS in the tree.
They all have a very specific misunderstanding. Go embeds _do_ support relative paths like:
//go:embed files/hello.txt
But they DO NOT support any paths with ".." in it
//go:embed ../files/hello.txt
is not correct.
All confidently claimed that .. is correct and will work and tried to make it work multipled different ways until I pointed each to the documentation.
Re: Context is the bottleneck for coding agents now
#183Earlier quoted context omitted.
Right, but the discussion we're having here is context size. I, and others, are saying that the current context size is a limitation on when they can use the tool to be useful. The replies of "well, just change the situation, so context doesn't matter" is irrelevant, and off-topic. The rationalizations even more so.
A huge context is a problem for humans too, which is why I think it's fair to suggest maybe the tool isn't the (only) problem. Tools like Aider create a code map that basically indexes code into a small context. Which I think is similar to what we humans do when we try to understand a large codebase. I'm not sure if Aider can then load only portions of a huge file on demand, but it seems like that should work pretty…
As someone who's worked with both more fragmented/modular codebases with smaller classes and shorter files vs ones that span thousands of lines (sometimes even double digits), I very much prefer the former and hate the latter.
That said, some of the models out there (Gemini 2.5 Pro, for example) support 1M context; it's just going to be expensive and will still probably confuse the model somewhat when it comes to the output.
Re: Context is the bottleneck for coding agents now
#184Earlier quoted context omitted.
Is there a good example you could provide of that? I just haven’t seen that personally so I’d be interested in any examples on these current models. I’m sure we all remember in the early days lots of examples of stupidity being posted and it was interesting. It be great if people kept doing that so we could get a better sense of which types of problems they are failing with astounding levels of stupidity on.
I've had every single LLM I tried (Opus, Sonnet, GPT-5-(codex) and Grok light) all tell me that Go embeds[0] support relative paths UPWARDS in the tree. They all have a very specific misunderstanding. Go embeds _do_ support relative paths like: //go:embed files/hello.txt But they DO NOT support any paths with ".." in it //go:embed ../files/hello.txt is not correct. All confidently claimed that .. is correct and will…
I don’t really find that so surprising or particularly stupid. I was hoping to learn about serious issues with bad logic or reasoning not missing dots on i’s type stuff.
I can’t remember the example but there was another frequent hallucination that people were submitting bug reports that it wasn’t working, so the project looked at it and realized well actually that kinda would make sense and maybe our tool should work like that, and changed the code to work just like the LLM hallucination expected!
Also in general remember human developers hallucinate ALL THE TIME and then realize it or check documentation. So my point is I feel hallucinations are not particularly important or bother me as much as flawed reasoning.
Re: Context is the bottleneck for coding agents now
#185Earlier quoted context omitted.
This is interesting, and I'd say you're not the target audience. If you want the code Claude writes to be line-by-line what you think is most appropriate as a human, you're not going to get it. You have to be willing to accept "close-ish and good enough" to what you'd write yourself. I would say that most of the time I spend with Claude is to get from its initial try to "close-ish and good enough". If I was working o…
The parent comment didn’t say anything about expecting the LLM output “to be line-by-line what you think is most appropriate as a human”?
If I were making a single line code change, then Claude's "style" would take me enough time to edit away that it would make it slower than writing the change myself. I'm positing this is true also for the parent commenter.
Re: Context is the bottleneck for coding agents now
#186I don't think intelligence is increasing. Arbitrary benchmarks don't reflect real world usage. Even with all the context it could possibly have, these models still miss/hallucinate things. Doesn't make them useless, but saying context is the bottleneck is incorrect.
Gemini 2.5 Pro is okay if you ask it to work on a very tiny problem. That's about it for me, the other models don't even create a convincing facsimile of reasoning.
Re: Context is the bottleneck for coding agents now
#187Re: Context is the bottleneck for coding agents now
#188Earlier quoted context omitted.
This is why I'm writing my own agent code instead of using simonw's excellent tools or just using Claude; the most interesting decisions are in the structure of the LLM loop itself, not in how many random tools I can plug into it. It's an unbelievably small amount of code to get to the point of super-useful results; maybe like 1500 lines, including a TUI.
And even if you do use Claude for actual work, there is also immense pedagogical value in writing an agent from scratch. Something really clicks when you actually write the LLM + tool calls loop yourself. I ran a workshop on this at my company and we wrote a basic CLI agent in only 120 lines of Python, with just three tools: list files, read file, and (over)write file. (At that point, the agent becomes capable enough…
Do you have this workshop available online? I’m really struggling to understand what “tool calls” and MCP are!
Re: Context is the bottleneck for coding agents now
#189Earlier quoted context omitted.
So you hold all that code context in your head at the same time? > If I need more, there is git, tickets, I can ask the person who wrote the code. What does this have to do with anything? Go ahead and ask the person. The notes the LLM writes aren’t for you they are for the LLM. You do you.
So you hold all that code context in your head at the same time? Yes. That is how every single piece of code has been writen since the creation of computers. Why you seem so surprised?
No reply? Probably because you've realized how much of an idiot you are?
Re: Context is the bottleneck for coding agents now
#190There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.
You don't want to discard prior information though. That's the problem with small context windows. Humans don't forget the original request as they ask for more information or go about a long task. Humans may forget parts of information along the way, but not the original goal and important parts. Not unless they have comprehension issues or ADHD, etc.
This isn't a misconception. Context is a limitation. You can effectively have an AI agent build an entire application with a single prompt if it has enough (and the proper) context. The models with 1m context windows do better. Models with small context windows can't even do the task in many cases. I've tested this many, many, many times. It's tedious, but you can find the right model and the right prompts for success.