Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

121–130 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#121

Post by _doctor_love »

dhorthy wrote: post #56
_doctor_love wrote: post #2

> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/ Seems like it went pretty well if a consulting company is now being starte…

> In order for coding with LLMs to go well, there has to be more rigor, more discipline, more good engineering hard-assedness. To reiterate, the teams seeing the best results with AI were already high-discipline and high-hygiene. hard agree. But i don't think this is sufficient. Even formal verification has its limitations. > AI works on data. The better the data, the better the likelihood of a desirable outcome. Cod…

[dead]

Re: Why Software Factories Fail (or: harness engineering is not enough)

#122

Post by satvikpendem »

dhorthy wrote: post #117
satvikpendem wrote: post #116

Earlier quoted context omitted.

They turned the lights off in 2025. The post is simply not accurate to today, they even say so, stating that apparently no one has proof of better models in 2026 improving codebases over time. I've seen it happen but the author hasn't apparently, so it's a game of "you said I said", not anything concrete.

(op here btw) - the question isn't "can models make code better" - its "left fully unattended, will they turn your codebase to slop over time" from the footnotes (sorry if this got a little buried) > yes of course you can get gpt-5.5 xhigh to do BRILLIANT refactors. But you had to tell it to do that. And to tell it to do that you had to understand your codebase well enough to know it needed doing. We're here talking…

GPT 5.6 Sol and Fable are similar step changes, not 5.5, but that's beside the point. If in the Markdown instruction files you ask them to periodically clean up the code or refactor where necessary then that's what they'll do. They don't explicitly do it because otherwise people, as they had for previous models, complain where models are asked to change one thing then they rewrite whole systems, so now the latest models are explicitly trained on instruction following.

In other words you can make the AI act like a software engineer complete with fixing and improving code autonomously (Cursor gave a glimpse of this [0]). It is not a failure of the model to do this automatically but a failure of the human having the wrong instructions.

[0] https://news.ycombinator.com/item?id=48982535

Re: Why Software Factories Fail (or: harness engineering is not enough)

#123

Post by zahrevsky »

jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

> why not do RL and make a penalty for bad design?

But this is clearly explained in the article. We can tell whether the design was good or bad only weeks or even months later, which is too long to wait for artificial trajectories. Tests, by contrast, provide validation in seconds or minutes.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#124

Post by stellar_jay »

_doctor_love wrote: post #29
stellar_jay wrote: post #7

Earlier quoted context omitted.

> Prompts are simply not enough to steer a coding agent to the level of precision needed. Without deep programmatic verification - at all levels, formal verification is just one slice - the solutions the agent produces will always be just slightly (or very) out of true. I found this to be exactly right, and in my work I’ve come up with a taxonomy of constraint mechanisms which I keep in mind when guiding agents: gene…

That's an excellent writeup. Haven't gotten all the way through it yet but so far I'm with you.

[dead]

Re: Why Software Factories Fail (or: harness engineering is not enough)

#125

Post by _doctor_love »

dhorthy wrote: post #72
claytonjy wrote: post #71

Earlier quoted context omitted.

> There are 'points of view' that emerge during coding I know it’s a bit cliche at this point, but this harkens to “programming as theory building”[0] which I agree is easy to lose out on when embracing agentic coding today. [0]: https://gwern.net/doc/cs/algorithm/1985-naur.pdf

I have had the phrase "programming is building a theory" spinning in my head for days, especially since watching this pragmatic engineer pod with Kent Beck https://www.youtube.com/watch?v=ddHQQtjIOpw

You're hopefully already aware that this references a famous paper from 1985? https://pages.cs.wisc.edu/~remzi/Naur.pdf

Re: Why Software Factories Fail (or: harness engineering is not enough)

#126

Post by Terr_ »

Fordec wrote: post #51
jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

I have a hunch it comes down to the training data. We have spent decades as an industry talking up the new shiny thing, deliverables, frameworks, features, performance improvements, the one thing you don't read copious amounts of prose about is "how I maintained our system so that nobody noticed".

"Here's me throwing code away because it's inherently bad or no longer suitable for the new purpose."

Re: Why Software Factories Fail (or: harness engineering is not enough)

#127

Post by satvikpendem »

rom1504 wrote: post #118
jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

I think the simple answer is LLM cannot do long term planning. Maintainability means thinking what will happen to this code in the next year and possibly hundreds of changes and based on that selecting the right abstractions that will work in the long term. LLMs currently cannot do long term planning and specifically cannot pick abstractions that will work in the long term. I think that's the important problem to sol…

Why can they not do long term planning? They know what good software design looks like and can code accordingly. Using Fable for a greenfield project for example, it doesn't dump everything into one file but instead does indeed split out the work into logical units as needed.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#128

Post by dhorthy »

satvikpendem wrote: post #122
dhorthy wrote: post #117

Earlier quoted context omitted.

(op here btw) - the question isn't "can models make code better" - its "left fully unattended, will they turn your codebase to slop over time" from the footnotes (sorry if this got a little buried) > yes of course you can get gpt-5.5 xhigh to do BRILLIANT refactors. But you had to tell it to do that. And to tell it to do that you had to understand your codebase well enough to know it needed doing. We're here talking…

GPT 5.6 Sol and Fable are similar step changes, not 5.5, but that's beside the point. If in the Markdown instruction files you ask them to periodically clean up the code or refactor where necessary then that's what they'll do. They don't explicitly do it because otherwise people, as they had for previous models, complain where models are asked to change one thing then they rewrite whole systems, so now the latest mod…

i guess to clarify my contention:

generic prompting like "review this code" or "make the architecture better" will raise the floor but cannot come close to human-quality code without humans understanding what code exists and what to ask for.

Obviously cursor and other labs have a stake in this going one direction, but I don't buy it, and I don't think you should either.

In fact, "Rebuild sqlite from spec" has all the problems with every other benchmark that I cited - model knows the whole problem up front and never has to iterate on the pile of slop it created cheating its way to a solution.

In any case, politely, I think we're mostly arguing vibes here and I'm not sure its going to get anywhere.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#129

Post by 20k »

satvikpendem wrote: post #111
dmurvihill wrote: post #108

Earlier quoted context omitted.

Wow, good to know. Can you tell me what company you work for so I can make sure it's not part of my stack?

It's many if not most companies now doing automated PR review.

Asserted confidently without a shred of proof

Re: Why Software Factories Fail (or: harness engineering is not enough)

#130

Post by 20k »

amannm wrote: post #42
20k wrote: post #22

Earlier quoted context omitted.

>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant The reason for human code review is: 1. So *you* understand what's going…

It seems weird to get angry about a change that is happening and will continue to happen due to what the market demands from software, which I believe will be the speed/ability to solve problems, rather than its own stewardship

I'm just here to do good software engineering and make good products. If people want to try and circumvent that process to deliberately make the product worse and slow development down, they're bad software engineers, its as simple as that

Nobody thinks its a good idea to fully automate PRs, at least not anyone with software engineering experience. It slows down your ability to solve problems if your code base is an absolute disaster

Again: If someone fully automates everything, what service do they provide? Why are they even there? Especially actively lying in the process when people are trying to force knowledge dissemination with human reviewed PRs

Post reply on HN