Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context is the bottleneck for coding agents now

runnercode.com

Post reply on HN
Print view View original

141–150 of 193 posts

Re: Context is the bottleneck for coding agents now

#141

Post by menaerus »

throwacct wrote: post #117

We stopped hiring a while ago because we were adjusting to "AI". We're planning to start hiring next year, as upper management finally saw the writing on the wall: LLMs won't evolve past junior engineers, and we need to train junior engineers to become mid-level and senior engineers to keep the engine moving. We're now using LLMs as mere tools (which is what it was meant to be from the get-go) to help us with differe…

> That sentiment will be the same for doctors, lawyers, etc., and personally, I won't put my life in the hands of any LLMs when it comes to finances, health, or personal well-being, for that matter.

I mean, did you try it for those purposes?

I have personally submitted an appeal to court for an issue I was having for which I would otherwise have to search almost indefinitely for a lawyer to be even interested into it.

I also debugged health opportunities from different angles using the AI and was quite successful at it.

I also experimented with the well-being topic and it gave me pretty convincing and mind opening suggestions.

So, all I can say is that it worked out pretty good in my case. I believe its already transformative in a ways we wouldn't be able even to envision couple years ago.

Re: Context is the bottleneck for coding agents now

#143

Post by sebastiennight »

atonse wrote: post #63
ray__ wrote: post #18

Earlier quoted context omitted.

This is a great insight. Any thoughts on how to address this problem?

Do we know if LLMs understand the concept of time? (like i told you this in the past, but what i told you later should supersede it?) I know there classes of problems that LLMs can't natively handle (like doing math, even simple addition... or spatial reasoning, I would assume time's in there too). There are ways they can hack around this, like writing code that performs the math. But how would you do that for chrono…

All it sees is a big blob of text, some of which can be structured to differentiate turns between "assistant", "user", "developer" and "system".

In theory you could attach metadata (with timestamps) to these turns, or include the timestamp in the text.

It does not affect much, other than giving the possibility for the model to make some inferences (eg. that previous message was on a different date, so its "today" is not the same "today" as in the latest message).

To chronologically fade away the importance of a conversation turn, you would need to either add more metadata (weak), progressively compact old turns (unreliable) or post-train a model to favor more recent areas of the context.

Re: Context is the bottleneck for coding agents now

#144

Post by jdrek1 »

AlGoreRhythm wrote: post #98
neutronicus wrote: post #24

Earlier quoted context omitted.

No, I think context itself is still an issue. Coding agents choke on our big C++ code-base pretty spectacularly if asked to reference large files.

Out of curiosity, how would you rate an LLM’s ability to deal with pointers in C++ code?

If you have lots of pointers, you're writing C, not C++.

Re: Context is the bottleneck for coding agents now

#145

Post by neutronicus »

atonse wrote: post #75
neutronicus wrote: post #24

Earlier quoted context omitted.

No, I think context itself is still an issue. Coding agents choke on our big C++ code-base pretty spectacularly if asked to reference large files.

I've found situations where a file was too big, and then it tries to grep for what might be useful in that file. I could see in C++ it getting smarter about first checking the .h files or just grepping for function documentation, before actually trying to pull out parts of the file.

Yeah, my first instinct has been to expose an LSP server as a tool so the LLM can avoid reading entire 40,000 line files just to get the implementation of one function.

I think with appropriate instructions in the system prompt it could probably work on this code-base more like I do (heavy use of Ctrl-, in Visual Studio to jump around and read only relevant portions of the code-base).

Re: Context is the bottleneck for coding agents now

#147

Post by sethhochberg »

DenisM wrote: post #136
sethhochberg wrote: post #121

Earlier quoted context omitted.

I've had great luck with asking the current session to "summarize our goals, conversation, and other relevant details like git commits to this point in a compact but technically precise way that lets a new LLM pick up where we're leaving off". The new session throws away whatever behind-the-scenes context was causing problems, but the prepared prompt gets the new session up and running more quickly especially if pick…

Wow, I had useless results asking “please summarize important points of the discussion” from ChatGPT. It just doesn’t understand what’s important, and instead of highlighting pivoting moments of the conversation it produce a high level introduction for a non-practitioner. Can you share you prompt?

Honestly, I just type out something by hand that is roughly like what I quoted above - I'm not big on keeping prompt libraries.

I think the important part is to give it (in my case, these days "it" is gpt-5-codex) a target persona, just like giving it a specific problem instead of asking it to be clever or creative. I've never asked it for a summary of a long conversation without the context of why I want the summary and who the intended audience is, but I have to imagine that helps it frame its output.

Re: Context is the bottleneck for coding agents now

#148

Post by KineticLensman »

PantaloonFlames wrote: post #130
raincole wrote: post #122

Earlier quoted context omitted.

Humans have a very strong tendency (and have made tremendous collective efforts) to compress context. I'm not a neuroscientist but I believe it's called "chunk." Language itself is a highly compressed form of compressed context. Like when you read "hoist with one's own petard" you don't just think about literal petard but the context behind this phrase.

We don’t think of petards because no one knows what that is. :)

For anyone wondering, it means blown into the air (‘hoist’) by your own bomb (‘petard’). From Shakespeare

Re: Context is the bottleneck for coding agents now

#149

Post by nowittyusername »

_joel wrote: post #36

I'm making a pretty complex project using claude. I tried claude flow and some other orchestrators but they produced garbage. Have found using github issues to track the progress as comments works fairly well, the PR's can get large comment wise (especially if you have gemini code assist, recommeded as another code review judge), so be mindful of that (that will blow the context window). Using a fairly lean CLAUDE.md…

your comment reminds me of another one i saw on reddit. someone said they found that using github diff as a way to manage context and reference chat history worked the best for their ai agent. i think he is on to something here.

Re: Context is the bottleneck for coding agents now

#150

Post by ericmcer »

bgirard wrote: post #31
aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

It seems possible for openAI/Anthropic to rework their tools so they discard/add relevant context on the fly, but it might have some unintended behaviors.

The main thing is people have already integrated AI into their workflows so the "right" way for the LLM to work is the way people expect it to. For now I expect to start multiple fresh contexts while solving a single problem until I can setup a context that gets the result I want. Changing this behavior might mess me up.

Post reply on HN