Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context is the bottleneck for coding agents now

runnercode.com

Post reply on HN
Print view View original

81–90 of 193 posts

Re: Context is the bottleneck for coding agents now

#81

Post by kypro »

I'm hitting 'x' to doubt hard on this one.

The ICPC is a short (5 hours) timed contest with multiple problems, in which contestants are not allowed to use the internet.

The reason most don't get a perfect score isn't because the tasks themselves are unreasonably difficult, but because they're difficult enough that 5 hours isn't a lot of time to solve so many problems. Additionally they often require a decent amount of math / comp-sci knowledge so if you don't know have the knowledge necessary you probably won't be able complete it.

So to get a good score you need lots of math & comp-sci knowledge + you need to be a really quick coder.

Basically the consent is perfect for LLMs because they have a ton of math and comp-sci knowledge, they can spit out code at super human speeds, and the problems themselves are fairly small (they take a human maybe 15 mins to an hour to complete).

Who knows, maybe OP is right and LLMs are smart enough to be super human coders if they just had the right context, but I don't think this example proves their point well at all. These are exactly the types of problems you would expect a supercharged auto-complete would excel at.

Re: Context is the bottleneck for coding agents now

#82

Post by loudmax »

atonse wrote: post #63
ray__ wrote: post #18

Earlier quoted context omitted.

This is a great insight. Any thoughts on how to address this problem?

Do we know if LLMs understand the concept of time? (like i told you this in the past, but what i told you later should supersede it?) I know there classes of problems that LLMs can't natively handle (like doing math, even simple addition... or spatial reasoning, I would assume time's in there too). There are ways they can hack around this, like writing code that performs the math. But how would you do that for chrono…

LLMs certainly don't experience time like we do. They live in a uni-dimensional world that consists of a series of tokens (though it gets more nuanced if you account for multi-modal or diffusion models). They pick up some sense of ordering from their training data, such as "disregard my previous instruction," but it's not something they necessarily understand intuitively. Fundamentally, they're just following whatever patterns happen to be in their training data.

Re: Context is the bottleneck for coding agents now

#83

Post by layer8 »

agf wrote: post #48
marstall wrote: post #9

> Level 2 - One commit - Cursor and Claude Code work well for tasks in this size range. I'll stop ya right there. Spending the past few weeks fixing bugs in a big multi-tier app (which is what any production software is this days). My output per bug is always one commit, often one line. Claude is an occasional help, nothing more. Certainly not generating the commit for me!

This is interesting, and I'd say you're not the target audience. If you want the code Claude writes to be line-by-line what you think is most appropriate as a human, you're not going to get it. You have to be willing to accept "close-ish and good enough" to what you'd write yourself. I would say that most of the time I spend with Claude is to get from its initial try to "close-ish and good enough". If I was working o…

The parent comment didn’t say anything about expecting the LLM output “to be line-by-line what you think is most appropriate as a human”?

Re: Context is the bottleneck for coding agents now

#84

Post by adastra22 »

cjbgkagh wrote: post #77
davedx wrote: post #71

Earlier quoted context omitted.

Yeah I start a new session to mitigate this. Don’t keep hammering away - close the current chat/session whatever and restate the problem carefully in a new one.

There should be a simple button that allows you refine the context. A fresh LLM could generate a new context from the input and outputs of the chat history, then another fresh LLM can start over with that context.

/compact in Claude Code.

Re: Context is the bottleneck for coding agents now

#85

Post by ISL »

If not now, soon, the bottleneck will be responsibility. Where errors in code have real-world impacts, "the agentic system wrote a bug" won't cut it for those with damages.

As these tools make it possible for a single person to do more, it will become increasingly likely that society will be exposed to greater risks than that single person's (or small company's) assets can cover.

These tools already accelerate development enough that those people who direct the tools can no longer state with credibility that they've personally reviewed the code/behavior with reasonable coverage.

It'll take over-extensions of the capability of these tools, of course, before society really notices, but it remains my belief that until the tools themselves can be held liable for the quality of their output, responsibility will become the ultimate bottleneck for their development.

Re: Context is the bottleneck for coding agents now

#86

Post by moffkalast »

dingnuts wrote: post #55
bgirard wrote: post #31

Earlier quoted context omitted.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

that's because a next token predictor can't "forget" context. That's just not how it works. You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want That's why they have a tendency to go ahead and do things you tell them not to do.. also IDK about yo…

That's not how attention works though, it should be perfectly able to figure out which parts are important and which aren't, but the problem is that it doesn't really scale beyond small contexts and works on a token to token basis instead of being hierarchical with sentences, paragraphs and sections. The only models that actually do long context do so by skipping attention layers or doing something without attention or without positional encodings, all leading to shit performance. Nobody pretrains on more than like 8k, except maybe Google who can throw TPUs at the problem.

Re: Context is the bottleneck for coding agents now

#87

Post by jofla_net »

dingnuts wrote: post #55
bgirard wrote: post #31

Earlier quoted context omitted.

I think that's the real issue. If the LLM spends a lot of context investigating a bad solution and you redirect it, I notice it has trouble ignoring maybe 10K tokens of bad exploration context against my 10 line of 'No, don't do X, explore Y' instead.

that's because a next token predictor can't "forget" context. That's just not how it works. You load the thing up with relevant context and pray that it guides the generation path to the part of the model that represents the information you want and pray that the path of tokens through the model outputs what you want That's why they have a tendency to go ahead and do things you tell them not to do.. also IDK about yo…

Relax friend! I can't see why you'd be peeved in the slightest! Remember, the CEOs have it all figured out and have 'determined' that we don't need all those eyeballs on the code anymore. You can simply 'feed' the machine and do the work of forty devs! This is the new engineering! /s

Re: Context is the bottleneck for coding agents now

#88

Post by binary132 »

In my opinion human beings also do not have unlimited cognitive context. When a person sits down to modify a codebase, they do not read every file in the codebase. Instead they rely on a combination of working memory and documentation to build the high-level and detailed context required to understand the particular components they are modifying or extending, and they make use of abstraction to simplify the context they need to build. The correct design of a coding LLM would require a similar approach to be effective.

Re: Context is the bottleneck for coding agents now

#89

Post by sheerun »

aliljet wrote: post #14

There's a misunderstanding here broadly. Context could be infinite, but the real bottleneck is understanding intent late in a multi-step operation. A human can effectively discard or disregard prior information as the narrow window of focus moves to a new task, LLMs seem incredibly bad at this. Having more context, but leaving open an inability to effectively focus on the latest task is the real problem.

Could be, but it's not. As soon as it will be infinite new brand of solutions will emerge

Re: Context is the bottleneck for coding agents now

#90

Post by bongodongobob »

Someone1234 wrote: post #42
neutronicus wrote: post #24

Earlier quoted context omitted.

No, I think context itself is still an issue. Coding agents choke on our big C++ code-base pretty spectacularly if asked to reference large files.

Yeah, I have the same issue too. Even for a file with several thousand lines, they will "forget" earlier parts of the file they're still working in resulting in mistakes. They don't need full awareness of the context, but they need a summary of it so that they can go back and review relevant sections. I have multiple things I'd love LLMs to attempt to do, but the context window is stopping me.

Interestingly, this issue has caused me to refactor and modularize code that I should have addressed a long time ago, but didn't have the time or stamina to tackle. Because the LLM can't handle the context, it has helped me refactor stuff (seems to be very good at this in my experience) and that has led me to write cleaner and more modular code that the LLMs can better handle.
Post reply on HN