Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

arxiv.org

Post reply on HN
Print view View original

201–210 of 230 posts

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#201

Post by coldtea »

sirsinsalot wrote: post #183
coldtea wrote: post #176

Earlier quoted context omitted.

> Not to be pedantic but the emergent properties are in the training set, and thus the model and algorithm. There's no magic coming from the universe. You are either being pedantic or missing the point of emergent however. Yes, it's not some novel unforeseen thing, like a magical Marvel Universe material or some unknown to humanity mode of thinking. Same way when people make something new they still recombine known w…

My point was that if training data + encoding/training = model with emergent behaviour The emergent behaviour is in the training data and/or encoding/training. So while I agree it is emergent from the complexity, it isn't some unknown mechanism. Just complexity at scale.

>So while I agree it is emergent from the complexity, it isn't some unknown mechanism. Just complexity at scale.

So like humans? Like the universe?

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#202

Post by coldtea »

dominotw wrote: post #179
coldtea wrote: post #154

Earlier quoted context omitted.

I don't think you understand how LLMs work. They're not merely re-arranging pre-existing blocks of code. And they have been shown to develop emergent properties that weren't in their training set time and again. They generate novel things as much as the average programmer (which works after himself having practice, exposure to codebases, and training, and reading API documentation, and such) generates novel things.

since you understand how llms really work. show us what novel items llms have generated for you.

How about a novel theorem escaping mathematicians for close to a century?

https://www.scientificamerican.com/article/ai-just-solved-an...

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#203

Post by Npovview »

ethin wrote: post #196
Npovview wrote: post #164

Earlier quoted context omitted.

If you have been following the news, harness is also a scaling direction now. Prompt your AI better not to forget relevant stuff or write them in a file which it can refer later. This way context can be refreshed, this is cached facts method or rolling window method of refreshing your memory just like you would ask a colleague to explain a concept again. These are solved problems.

Are they though? Because I really shouldn't have to use Claude Code (and I don't) just to get even decent results. As I said, I thought one of the biggest advantages AI was supposed to have was that it wouldn't need such constant reminding of things because it wasn't trying to emulate us faulty, forgetful, fragile humans who do have memory loss?

You can convert your best practices into a skill or best practices md file and CC will keep that in purview.

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#204

Post by antonvs »

leecommamichael wrote: post #191
antonvs wrote: post #50

Earlier quoted context omitted.

Critiques like this tend to focus very hard on what models can't do. It's true, they have limitations. But they're also superhuman in so many other ways. It's valid to point out limitations, but that doesn't support the conclusion that models are not incredibly powerful and capable of the functional equivalent of reasoning at human or superhuman levels in many scenarios.

Agreed. Both are true. I sometimes think of the calculator as being superhuman as well.

Yes, although the calculator couldn't "reason" the way ML models can.

All the political and emotional reactions to LLMs seem to obscure how absolutely amazing this technology is. I've pointed them at codebases I wrote entirely myself and had them find bugs, point things out I had missed, plan and implement refactorings to improve code quality, etc. I may be "smarter" than the models in some ways but there's no question they're smarter than me in others. They're unlike any tool we've ever had access to.

Yes, the politics and economics around them leaves a lot to be desired (read: is absolutely terrible), and there are a lot of valid justifications for the "AI backlash", but there's a very important baby in that bathwater.

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#205

Post by wetpaws »

rrook wrote: post #57

As a codebase grows, divergent structural emergence from incidental(lang and lib) details results in prolonged complexity costs. I'm working on a language that enforces structure for agents: https://github.com/hale-lang/hale

[dead]

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#206

Post by AIorNot »

Why the heck does this have to be a scientific study? since when did SWEs publish archive style science peices.. lol a good blog post would have been better.

LLMs write working code, but have trouble following the script. They are slot machines of code. Human oversight is under pressure to deliver faster but code takes time to comprehend and analyze. Also in LLM coding, we end up with lots of natural language based spec files to manage and code we don't have an intuitive feel for unless we commit to the rigor of deep code review..(which no human really does anyway)

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#207

Post by hansmayer »

guhcampos wrote: post #93

I'm a convert. I was 100% skeptical about LLM code generation, now over 80% of the professional code I write is generated. That said, the limitations are kind of obvious and are starting to show in some of my projects, and this article seems to confirm my suspicions. If it's just confirmation bias or not, I can't say yet. In my experience, for anything complex enough, I have to start adding more and more constraints,…

> I have to start adding more and more constraints, style guides, corner cases, error handling, optimization guidelines and all this good stuff to my Markdown specifications, rules and skills

So kind of like maintaining a growing codebase? But this time around you cannot guarantee what the outputs will be?

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#208

Post by hansmayer »

tcoff91 wrote: post #140
sdevonoes wrote: post #138

Earlier quoted context omitted.

> At some point this starts to look like we're all just moving complexity from the more formal and deterministic world of programming languages to the informal and non-deterministic world of natural language. This is the problem nobody is talking about. I see codebases growing in MD files with instructions and guidelines and requests that are also LLM generated… and it’s all piling up. No one is reviewing it 100% , a…

This is why you need to be generating more linter rules instead of just having things be in markdown files. I had never written an eslint rule until i started having agents pump them out for me and now I've encoded a bunch of important rules as lint rules that will fail CI if violated.

A linter won't prevent your idiot LLM from going bonkers and suddenly switching to GQL instead of REST just for that one endpoint, because it confabulated something or putting your stripe secret into your react frontend - all cases of slop I've seen happen.

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#209

Post by hansmayer »

bitexploder wrote: post #190
tpmoney wrote: post #188

Earlier quoted context omitted.

One question I have is are these "constraints, style guides, corner cases, error handling, optimization guidelines" extra things that you wouldn't need otherwise, or are they formal documentation of the baked in assumptions and knowledge accumulated over the years? Every project I've ever worked on has had heaps of shared knowledge that's just part of stuff the team just "knows" and no one ever really writes down. Th…

There is a great thing. Because the agents can do so much toil you can add things like formal verification, fuzzing, and other feedback mechanisms and quality gates to your projects cheaply. In a human written project you still needed those things, but it cost a lot. Agents require these quality gates and they can implement them for you. The problem with AI documentation is it will just write a lot of useless bullshi…

> Because the agents can do so much toil you can add things like formal verification, fuzzing, and other feedback mechanisms and quality gates to your projects cheaply

Works great until they sweep you a test under the rug which always passes because the condition is something like if(true) .

Re: Constraint Decay: The Fragility of LLM Agents in Back End Code Generation

#210

Post by hansmayer »

coldtea wrote: post #155
archagon wrote: post #118

Earlier quoted context omitted.

That’s why we abstract the useful code away as libraries, frameworks, etc. AI is not an abstraction.

Of course it is. It abstracts away the code generation to a much more compressed natural language prompt. That's the very definition of a high abstraction...

It abstracts shit mate. How many Rs in the strawberry and if you want to go to a car wash, should you walk, run or drive? They're just fucking text generators, which happen to spit out something half-usable about 60% of the time.
Post reply on HN