Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

111–120 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#111

Post by satvikpendem »

dmurvihill wrote: post #108
4lx87 wrote: post #18

Earlier quoted context omitted.

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

Wow, good to know. Can you tell me what company you work for so I can make sure it's not part of my stack?

It's many if not most companies now doing automated PR review.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#112

Post by satvikpendem »

jeremyjh wrote: post #99
4lx87 wrote: post #18

Earlier quoted context omitted.

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

This entire article is about the fact that the codebase deteriorates over time when you work this way. If they are correct, it will take longer and longer to develop PRs, and more and more cycles to QA them. Eventually the code will be unsalvageable and it might get there before anyone notices a problem. I don’t know if they are right but I know codebases can get there with humans at the helm - and the humans can ada…

The bet is that models get better faster than code gets worse. Looks like so far it's correct, as Fable can already write better code than me and most others on average.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#113

Post by satvikpendem »

20k wrote: post #22
4lx87 wrote: post #18

Earlier quoted context omitted.

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant The reason for human code review is: 1. So *you* understand what's going…

Or that their company does not in fact enforce human PR reviews or even encourages AI PR reviews.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#114

Post by dhorthy »

satvikpendem wrote: post #112
jeremyjh wrote: post #99

Earlier quoted context omitted.

This entire article is about the fact that the codebase deteriorates over time when you work this way. If they are correct, it will take longer and longer to develop PRs, and more and more cycles to QA them. Eventually the code will be unsalvageable and it might get there before anyone notices a problem. I don’t know if they are right but I know codebases can get there with humans at the helm - and the humans can ada…

The bet is that models get better faster than code gets worse. Looks like so far it's correct, as Fable can already write better code than me and most others on average.

the thesis of the post is that this is not true. fable can solve more problems but for complex systems it is not sufficiently diligent that you can turn the lights off.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#115

Post by xorcist »

lozenge wrote: post #92
xorcist wrote: post #12

Earlier quoted context omitted.

> group file changes together based on theme, add some commentary Isn't that what commits are? Or ... should be?

No, commits always display the files in a fixed order, and then display changes from line 1 to line N. An AI could select the order to display in, add per hunk commentary and automatically adjust how much context lines are displayed.

But if you need to split the commit in hunks, and add a commentary per hunk, isn't that just a sign that you really should just split your commits?

That's the most killer feature of git, that it's so easy to slice your commits any way you desire, and then redo again. The use case of taking a chunk and commit separately is so common it even got a special mode in the add command. That, and the super fast jumping between branches is what set it apart from contemporary version control systems.

The extra context provided by the review tool is gone when the review is done anyway. Review systems come and go, but the commit log is for eternity.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#116

Post by satvikpendem »

dhorthy wrote: post #114
satvikpendem wrote: post #112

Earlier quoted context omitted.

The bet is that models get better faster than code gets worse. Looks like so far it's correct, as Fable can already write better code than me and most others on average.

the thesis of the post is that this is not true. fable can solve more problems but for complex systems it is not sufficiently diligent that you can turn the lights off.

They turned the lights off in 2025. The post is simply not accurate to today, they even say so, stating that apparently no one has proof of better models in 2026 improving codebases over time. I've seen it happen but the author hasn't apparently, so it's a game of "you said I said", not anything concrete.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#117

Post by dhorthy »

satvikpendem wrote: post #116
dhorthy wrote: post #114

Earlier quoted context omitted.

the thesis of the post is that this is not true. fable can solve more problems but for complex systems it is not sufficiently diligent that you can turn the lights off.

They turned the lights off in 2025. The post is simply not accurate to today, they even say so, stating that apparently no one has proof of better models in 2026 improving codebases over time. I've seen it happen but the author hasn't apparently, so it's a game of "you said I said", not anything concrete.

(op here btw) - the question isn't "can models make code better" - its "left fully unattended, will they turn your codebase to slop over time"

from the footnotes (sorry if this got a little buried)

> yes of course you can get gpt-5.5 xhigh to do BRILLIANT refactors. But you had to tell it to do that. And to tell it to do that you had to understand your codebase well enough to know it needed doing. We're here talking about why lights-off wont work

i suppose this would also be a good place to ask if you fall in the "high stakes production systems" camp

> If you love vibe coding, please, go on vibing. I still vibe code lots of things, I just also maintain lots of production software (and through HumanLayer, help 1000s of other engineers do the same), so the rest of this is aimed at folks solving hard problems in complex codebases.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#118

Post by rom1504 »

jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

I think the simple answer is LLM cannot do long term planning.

Maintainability means thinking what will happen to this code in the next year and possibly hundreds of changes and based on that selecting the right abstractions that will work in the long term.

LLMs currently cannot do long term planning and specifically cannot pick abstractions that will work in the long term.

I think that's the important problem to solve. Teaching LLMs how to pick the right abstractions.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#119

Post by goyozi »

dhorthy wrote: post #39
jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

fair point, this is the thing I struggled most to extract out while writing it - if you can propose an RL environment that penalizes a model for bad design, then I'm all ears - right now there's no fast oracle/verifier for this (as stated in the post) My current evolving take on "how would you build such a thing" is you need to tee up a roadmap of 20 features and feed them to a model one at a time, so it can't design…

I’ve experimented a bit with this approach for quasi-benchmarking models. Most models nowadays will keep trying until you cut them off, they’re happy to go into millions of tokens for an easy task, digging deeper and deeper into crap design. Compared to a clean reference implementation you get easily 20x token difference at task 11+ for the same model. Somewhat ironically I never published the results because the both the harness and the task set were vibe coded and I didn’t have the energy to clean it all up.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#120

Post by jdlshore »

jadar wrote: post #91
jdlshore wrote: post #82

Earlier quoted context omitted.

I think it’s because maintainability is a factor of software design, and good design is squishy and hard to articulate. There’s no good metrics for design. All the tools that purport to measure it are doing trivial things like cyclomatic complexity, and can only tell you when things have gotten really really bad, but not distinguish between “good” and “pretty bad.” Design skills take about 10 years to develop, in my…

> It’s just not stuff LLMs are good at. But then you're back at the question from the OP of "why not?" Is it lack of good examples in the training data? Is there something of code "goodness" that goes beyond phenomenon that are measurable? Is it that LLMs lack in the ability to organize and think abstractly?

All of the above, but mostly the latter, IMO.
Post reply on HN