Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

171–180 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#171
I’ve been postulating that knowledge of old school principles like Method One along with test driven development and domain-driven design are the external factors allowing a subset of teams to succeed.

Absent those skills, the agile movement is flailing with GenAI.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#172

Either you need to understand how your codebase works or you don’t. Claude can write the code for you but it can’t understand it for you. That part has to happen at human speeds. There are cases where you don’t have to understand everything, but I think that’s a more nuanced question. All of the above is true even if Claude writes perfect code.

My experience conflicts with this assertion. I've used Claude to achieve an understanding of two large codebases (that I mostly wrote, and certainly came up with most of the concepts therein) to the point that it's far superior to my understanding. I now get it to explain things to me that I have long forgotten.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#173
post #21
post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

PRs sucked to review long before Agents were a thing, but now it really sucks because there are more to review.

Right on. In many ways AI has just exposed how bogus and delusional our industry's "process" has been for years.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#174
post #22

Earlier quoted context omitted.

>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant The reason for human code review is: 1. So *you* understand what's going…

There are also compliance reasons. SOX requires code review by a second party. If the reviews are automated, the development process isn't in compliance and that can get expensive in a hurry.

And of course these SOX-mandated second reviews were 99% of the time pure theater.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#175

I find it amusing that people who are talking about Dark Software Factories, are talking about productivity in terms of number of pull requests or commits as a unit. If we are going in the Dark Software Factory route, why aren't we calling the code units as bos (bunch of shit) yet.

Surely "pile of...?"

Re: Why Software Factories Fail (or: harness engineering is not enough)

#176

Earlier quoted context omitted.

GPT 5.6 Sol and Fable are similar step changes, not 5.5, but that's beside the point. If in the Markdown instruction files you ask them to periodically clean up the code or refactor where necessary then that's what they'll do. They don't explicitly do it because otherwise people, as they had for previous models, complain where models are asked to change one thing then they rewrite whole systems, so now the latest mod…

i guess to clarify my contention: generic prompting like "review this code" or "make the architecture better" will raise the floor but cannot come close to human-quality code without humans understanding what code exists and what to ask for. Obviously cursor and other labs have a stake in this going one direction, but I don't buy it, and I don't think you should either. In fact, "Rebuild sqlite from spec" has all the…

And what percentage of the software in the world needs to come close to human-quality code? I’d argue the percentage of the whole is VERY low.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#177
post #34

I think there is a fundamental issue here of what building software even means If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "W…

If you're working with the 1% of human developers who think like that, sure.

My experience has been that current models are way beyond all but the most highly competent developers I've encountered in my several decades career.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#178

There's some good ideas and points in here, but this bit threw me: > # We tried this > In July 2025 we went full lights-off Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before. I feel like any perspective/experience on "what agents can/can't do" from be…

I had a bit of this impression when reading the post as well as the authors' product website. A lot of it does seem to be stuck in 2025. For instance I think their post "long-context isn't the answer" on their website post straight-up isn't accurate, and gives me the impression they are just extrapolating previous performance to new models. In my experience, Opus 4.6 and newer have worked very reliably for long conte…

> i don't perceive any intelligence drop at 700-900K tokens

I have countless examples of this for Fable 5 and Opus 4.8 at just half that many tokens. Sometimes they can't even remember the correct workflow even if they're merely an orchestrator and have been going through the same 5 step loop multiple times for pretty much their entire context window.

Wall of shame with an Opus and a Fable oopsie from just yesterday: https://i.imgur.com/znSJ3OE.png and https://i.imgur.com/DLOHQ1B.png

One of them is particularly egregious because it uses a CLI tool to step through the stages that literally tells it what to do next. How do you fuck this up? Needless to say that tool was revised to force the agents to acknowledge everything happened, but I'm sure it will find a way to eventually mess up anyhow.

And orchestration should be the easy part. It's pretty much just writing and adjusting workflow files. Don't get me started on code quality - but I realize that's subjective and my bar is unusually high. There's a reason I have them run adversarial verifiers along multiple axis before their output is even worth my attention.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#179

Earlier quoted context omitted.

> If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. I would rather claim that for my personal/hobby software, I apply a much higher quality standard than for the software that I…

You either need to quit or keep your head down, I can’t tell which.

If you need to actively keep your head down, you need to quit.

Doesn’t matter whether ‘the problem’ is the employer, the employee, both, or neither. They’re a bad fit for each other.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#180

Earlier quoted context omitted.

Agents are also controllable to a degree. How agents author PRs is now an engineering problem.

But this is my point, they're not really controllable in this way. There are no fixed rules (like tests or linters) for small PRs, it's not good enough to do it by line count. A good PR tells a story to the reader that leaves them confident in the result. That means knowing the reader and how they think, and that is something LLMs are very bad at.

yes well said
Post reply on HN