Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context is the bottleneck for coding agents now

runnercode.com

Post reply on HN
Print view View original

151–160 of 193 posts

Re: Context is the bottleneck for coding agents now

#151

Post by pessimizer »

menaerus wrote: post #141
throwacct wrote: post #117

We stopped hiring a while ago because we were adjusting to "AI". We're planning to start hiring next year, as upper management finally saw the writing on the wall: LLMs won't evolve past junior engineers, and we need to train junior engineers to become mid-level and senior engineers to keep the engine moving. We're now using LLMs as mere tools (which is what it was meant to be from the get-go) to help us with differe…

> That sentiment will be the same for doctors, lawyers, etc., and personally, I won't put my life in the hands of any LLMs when it comes to finances, health, or personal well-being, for that matter. I mean, did you try it for those purposes? I have personally submitted an appeal to court for an issue I was having for which I would otherwise have to search almost indefinitely for a lawyer to be even interested into it…

They're tuned (and its part of their nature) to be convincing to people who don't already know the answer. I couldn't get it to figure out how to substitute peanut butter for butter in a cookie recipe yesterday.

I ended up spending an hour on it and dumping the context twice. I asked it to evaluate its own performance and it gave itself a D-. It came up with the measurements for a decent recipe once, then promptly forgot it when asked to summarize.

Good luck trying to use them as a search engine (or a lawyer), because they fabricate a third of the references on average (for me), unless the question is difficult, then they fabricate all of them. They also give bad, nearly unrelated references, and ignore obvious ones. I had a case when talking about the Mexican-American war where the hallucinations crowded out good references. I assume it liked the sound of the things it made up more than the things that were available.

edit: I find it baffling that GPT-5 and Quen3 often have identical hallucinations. The convergence makes me think that there's either a hard limit to how good these things can get which has been reached, or that they're just directly ripping each other off.

Re: Context is the bottleneck for coding agents now

#152

Post by libraryofbabel »

tptacek wrote: post #108
libraryofbabel wrote: post #106

Earlier quoted context omitted.

Yes! - and I wish this was easier to do with common coding agents like Claude Code. Currently you can kind of do it manually by copying the results of the context-busting search, rewinding history (Esc Esc) to remove the now-useless stuff, and then dropping in the results. Of course, subagents are a good solution here, as another poster already pointed out. But it would be nice to have something more lightweight and…

This is why I'm writing my own agent code instead of using simonw's excellent tools or just using Claude; the most interesting decisions are in the structure of the LLM loop itself, not in how many random tools I can plug into it. It's an unbelievably small amount of code to get to the point of super-useful results; maybe like 1500 lines, including a TUI.

And even if you do use Claude for actual work, there is also immense pedagogical value in writing an agent from scratch. Something really clicks when you actually write the LLM + tool calls loop yourself. I ran a workshop on this at my company and we wrote a basic CLI agent in only 120 lines of Python, with just three tools: list files, read file, and (over)write file. (At that point, the agent becomes capable enough that you can set it to modifying itself and ask it to add more tools!) I think it was an eye-opener for a lot of people to see what the core of these things looks like. There is no magic dust in the agent; it's all in the LLM black box.

I hadn't considered actually rolling my own for day-to-day use, but now maybe I will. Although it's worth noting that Claude Code Hooks do give you the ability to insert your own code into the LLM loop - though not to the point of Eternal Sunshining your context, it's true.

Re: Context is the bottleneck for coding agents now

#153

Post by asadotzler »

menaerus wrote: post #141
throwacct wrote: post #117

We stopped hiring a while ago because we were adjusting to "AI". We're planning to start hiring next year, as upper management finally saw the writing on the wall: LLMs won't evolve past junior engineers, and we need to train junior engineers to become mid-level and senior engineers to keep the engine moving. We're now using LLMs as mere tools (which is what it was meant to be from the get-go) to help us with differe…

> That sentiment will be the same for doctors, lawyers, etc., and personally, I won't put my life in the hands of any LLMs when it comes to finances, health, or personal well-being, for that matter. I mean, did you try it for those purposes? I have personally submitted an appeal to court for an issue I was having for which I would otherwise have to search almost indefinitely for a lawyer to be even interested into it…

You are not a doctor, lawyer, etc. You are responsible for yourself, not for others like doctors and lawyers who face entirely different consequences for failures.

Re: Context is the bottleneck for coding agents now

#154

Post by reissbaker »

bgirard wrote: post #31
aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

IMO specifically OpenAI's models are really bad at being steered once they've decided to do something dumb. Claude and OSS models tend to take feedback better.

GPT-5 is brilliant when it oneshots the right direction from the beginning, but pretty unmanageable when it goes off the rails.

Re: Context is the bottleneck for coding agents now

#155

Post by Mikhail_Edoshin »

dingnuts wrote: post #55
bgirard wrote: post #31

Earlier quoted context omitted.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

that's because a next token predictor can't "forget" context. That's just not how it works. You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want That's why they have a tendency to go ahead and do things you tell them not to do.. also IDK about yo…

Well, "a sufficiently advanced technology is indistinguishable from magic". It's just that it is same in a bad way, not a good way.

Re: Context is the bottleneck for coding agents now

#156

Post by thunky »

Someone1234 wrote: post #138
thunky wrote: post #134

Earlier quoted context omitted.

It may be a good idea to refactor even if not for LLMs but for humans sake.

Right, but the discussion we're having here is context size. I, and others, are saying that the current context size is a limitation on when they can use the tool to be useful. The replies of "well, just change the situation, so context doesn't matter" is irrelevant, and off-topic. The rationalizations even more so.

A huge context is a problem for humans too, which is why I think it's fair to suggest maybe the tool isn't the (only) problem.

Tools like Aider create a code map that basically indexes code into a small context. Which I think is similar to what we humans do when we try to understand a large codebase.

I'm not sure if Aider can then load only portions of a huge file on demand, but it seems like that should work pretty well.

Re: Context is the bottleneck for coding agents now

#157

Post by hadlock »

Someone1234 wrote: post #42
neutronicus wrote: post #24

Earlier quoted context omitted.

No, I think context itself is still an issue. Coding agents choke on our big C++ code-base pretty spectacularly if asked to reference large files.

Yeah, I have the same issue too. Even for a file with several thousand lines, they will "forget" earlier parts of the file they're still working in resulting in mistakes. They don't need full awareness of the context, but they need a summary of it so that they can go back and review relevant sections. I have multiple things I'd love LLMs to attempt to do, but the context window is stopping me.

I've started getting in the habit of finding seams in files > 1500 lines long. Occasionally it is unavoidable, but very regularly.

Re: Context is the bottleneck for coding agents now

#158

Post by ninetyninenine »

hirako2000 wrote: post #23
ninetyninenine wrote: post #6

Context is a bottleneck for humans as well. We don’t have full context when going through the code because we can’t hold full context. We summarize context and remember summarizations of it. Maybe we need to do this with the LLM. Chain of thought sort of does this but it’s not deliberate. The system prompt needs to mark this as a deliberate task of building summaries and notes notes of the entire code base and this s…

That is not how the brain does it. We do take notes, we summarize our writings, that's a process. But the brain does not follow that primitive process to "scale".

We do. It’s just the format of what you remember is not textual. Do you remember what a 500 line function does or do you remember a fuzzy aspect of it?

You remember a fuzzy aspect of it and that is the equivalent of a summary.

The LLM is in itself a language machine so its memory will also be language. We can’t get away from that. But that doesn’t mean the hierarchical structure of how it stores information needs to be different from humans. You can encode information in anyway you like and store that information in any hierarchy we like.

So essentially We need the hierarchical structure of the “notes” that takes on the hierarchical structure of your memory. You don’t even access all your memory as a single context. You access parts of it. Your encoding may not be based on a “language” but for an LLM it’s basically a model based on language so its memory must be summaries in the specified language.

We don’t know every aspect of human memory but we do know the mind doesn’t access all memory at the same time and we do know that it compresses context. It doesn’t remember everything and it memorizes fuzzy aspects of everything. These two aspects can be replicated with the LLM entirely with text.

Re: Context is the bottleneck for coding agents now

#159

Post by ninetyninenine »

lomase wrote: post #116
ninetyninenine wrote: post #32

Earlier quoted context omitted.

I’m not talking about git diffs. I’m talking about the summaries of context. Every commit the ai needs to update the summaries and notes it took about the code. Did you read the entirety of what I wrote? Please read. Say the AI left a 5 line summary of a 300 line piece of code. You as a human update that code. What I am saying specifically is this: when you do the change, The AI then sees this and updates the summary…

The context is the code I work on because I can read and understand it. If I need more, there is git, tickets, I can ask the person who wrote the code. I do have read your comment, don't make snarky comments.

So you hold all that code context in your head at the same time?

> If I need more, there is git, tickets, I can ask the person who wrote the code.

What does this have to do with anything? Go ahead and ask the person. The notes the LLM writes aren’t for you they are for the LLM. You do you.

Re: Context is the bottleneck for coding agents now

#160

Post by ninetyninenine »

anthonypasq wrote: post #52
ninetyninenine wrote: post #6

Context is a bottleneck for humans as well. We don’t have full context when going through the code because we can’t hold full context. We summarize context and remember summarizations of it. Maybe we need to do this with the LLM. Chain of thought sort of does this but it’s not deliberate. The system prompt needs to mark this as a deliberate task of building summaries and notes notes of the entire code base and this s…

youre projecting a deficiency of the human brain onto computers. computers have advantages that our brains dont (perfect and large memory), theres no reason to think that we should try to recreate how humans do things. why would you bother with all these summaries if you can just read and remember the code perfectly.

Because the context window of the LLM is limited similar to humans. That’s the entire point of the article. If the LLM has similar limitations to humans than we give it similar work arounds.

Sure you can say that LLMs have unlimited context, but then what are you doing in this thread? The title on this page is saying that context is a bottleneck.

Post reply on HN