Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

arxiv.org

Post reply on HN
Print view View original

141–150 of 230 posts

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#141

Post by sdevonoes »

somewhatgoated wrote: post #128
mike_hock wrote: post #125

Earlier quoted context omitted.

> moving complexity from the more formal and deterministic world of programming languages to the informal and non-deterministic world of natural language It's like using a compiler that generates semantically different code every time you run it. Basically like compiling a program that's full of UB but "seems to work" most of the time. > business sees this as productivity gains Back to LoC/s as a measure of "producti…

> Back to LoC/s as a measure of "productivity." IMO this doesn’t follow from what OP wrote. I personally measure it with a more abstract “how long does it take me to ship something that is useful in production and solving a real problem” and the increase in speed there has been massive for me. But of course I’m not a bigbrain 10x coder that is doing bleeding edge novel stuff like most people here, so gains might be m…

> how long does it take me to ship something that is useful in production and solving a real problem

But that’s only half of the problem. What about “and how easy it is to maintain long-term”. If you say that maintenance can be done via LLM, I would argue that there is zero guarantees that LLMs are backwards compatible and that the markdown you wrote now will work just as fine in 1,2,3 years

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#142

Post by haeseong »

xcjsam wrote: post #25
bob1029 wrote: post #15

> Our findings reveal a phenomenon of constraint decay: as structural requirements accumulate, agent performance exhibits a substantial decline. I have exactly the inverse findings on my end. The bigger and more legacy the codebase, the more accurate the patches become. The harness itself seems to be the most important part. I use a recursive loop that primes the root context based on the user prompt each time. My ag…

The harness mattering more than the model lines up with my experience too. What this paper measures is within-turn constraint decay. The version that bites in multi-agent setups is across-session — the architectural rules an agent wrote down on Monday don't reach the agent making the next change on Tuesday.

[flagged]

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#145

Post by withinboredom »

dnautics wrote: post #96
withinboredom wrote: post #92

Earlier quoted context omitted.

Saying that it’s dismissive is like saying writing (insert language) is dismissive that you’re just writing assembly. at the end of the day, it presents a vector field and predicts the next vector. That’s literally the heart of intelligence just like assembly is the heart of execution. When playing table tennis, your brain is literally predicting seconds into the future to get your body into the right position. But w…

You're making my point for me, saying table tennis is "just a proprioceptive predictor" is dismissively reductive (and not a particularly useful framework for understanding table tennis), even if it is strictly speaking accurate. It's the sort of thing someone who has no idea how hard training for table tennis is would say.

Let me put it bluntly. I’m agreeing with you but saying that isn’t what I was talking about and trying to give examples. You’re also agreeing with me.

The “idea” of table tennis and the rules. Those are things we can talk about. It’s those “best practices” I gave in my example. The actual playing of table tennis would be the examples. How to apply those best practices and what good code looks like.

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#146

Post by Geezus_42 »

dominotw wrote: post #109
marcosdumay wrote: post #105

Earlier quoted context omitted.

You just threw the "easily" away from the comment you are replying.

doesnt make a difference to my comment

There is a huge difference between "not verifiable" and "not easily verifiable".

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#147

Post by sanxiyn »

abujazar wrote: post #70
spacedoutman wrote: post #37

This research is useless and nearly all other LLM research is too. gpt 5.2 is the strongest model they tested, a nearly 6 month old model. Traditional research can not keep up.

Agreed. As Simon Willison points out, November 2025 was a a critical months because that's pretty much when coding agents became «good enough», eliminating most of the problems pointed out in this study.

GPT-5.2 was released after November 2025.

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#148

Post by lemax »

Would love to see this benchmark tested on more perceivably LLM friendly frameworks/ORM (e.g. is NestJS or Drizzle / Kysely more performant than their choice of Sequelize) and more frontier model vs just GPT 5.2.

Anyone read whether these tests include any validation loops? What happens if the models get back test failures, for instance? Understanding how many turns to hit full passing behavior suite would also be interesting. Great methodology in the study though.

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#150

Post by anygivnthursday »

abujazar wrote: post #70
spacedoutman wrote: post #37

This research is useless and nearly all other LLM research is too. gpt 5.2 is the strongest model they tested, a nearly 6 month old model. Traditional research can not keep up.

Agreed. As Simon Willison points out, November 2025 was a a critical months because that's pretty much when coding agents became «good enough», eliminating most of the problems pointed out in this study.

I regularly see Claude Opus 4.7 dropping constraints from an otherwise small CLAUDE.md at merely 20% context use. I have to keep reminding it, and it has all info ready in its context, still time to time decides to ignore parts.
Post reply on HN