Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

61–70 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#61

> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/ Seems like it went pretty well if a consulting company is now being starte…

Some more context on the consulting company: StrongDM was sold earlier this year, about a year after the dark factory was first announced, and the former CTO moved on to this (presumably) in order to continue the idea. Disclaimer: I'm a former StrongDM employee

apg?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#62

> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/ Seems like it went pretty well if a consulting company is now being starte…

Hi, I'm one of the trio from the StrongDM AI Lab. Just a minor thing I want to clarify about the Weather Report [1] - it's framed in kind of a negative light in the article ("sparse updates"), but we've been updating it as frequently as we find a meaningful improvement in a relevant dimension. Since we launched it in February it has averaged about one update per month, as frontier labs keep racing forward! [1] https:…

appreciate that context! I definitely did not mean to come out and say "its definitely not working" or anything, but would love to hear from y'all a retrospective on the ~5-6 month anniversary - what was right, what did we get wrong, etc

Re: Why Software Factories Fail (or: harness engineering is not enough)

#63
post #38

My radical opinion is that LLMs are harmful for software development - they are the ultimate "goto" operator. All actual code should be written by a human developer. Instead, use them in adversarial mode - run QA scenarios using LLM agent as a substitute for end user to do bug discovery.

why?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#64
post #32

I've recently started experimenting with grounding LLM driven implementation/verification on RFC based normative specifications, to avoid having to manually steer the LLM during implementation and dealing with reviewing sloppy pull requests. It works quite well, as it puts your entire focus on writing (hopefully) unambiguous specifications vs. having to discuss unwanted changes with an LLM during code-review. One fla…

normative specifications can help, but the thesis here is that specs that define behavior of the product or even architecture are helpful but there's MORE that can be done and even though "program design" feels too in the weeds it's still essential if you care about maintainability

Isn't maintainability mostly about applying basic engineering principles e.g. separation of concerns, single responsibility, dependency inversion, open-closed etc..? If these topics are addressed in the normative representation, and correctly translated by an LLM into code (especially by slicing up the specifications into measurable outcomes to set the intended foundation), then future change e.g. maintainability becomes essentially easy as well, no?

Historically I know that the majority maintenance problems occur from slow continuous evolution of a system that it initially was never designed for. And the only way to address this was continuous system design.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#65
post #22
post #18

Earlier quoted context omitted.

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant The reason for human code review is: 1. So *you* understand what's going…

this is exactly right

Re: Why Software Factories Fail (or: harness engineering is not enough)

#66
post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

I think the right shape is to review and merge directly from the agent window.

reviewing code as it's being written & re-steering it > reviewing it once 20k lines have been written

Re: Why Software Factories Fail (or: harness engineering is not enough)

#67
post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

thanks for pointing out the linear PR stuff, hadn’t seen that. Interesting that while a dozen other companies are trying to muscle in on the hosting/versioning side of github, rather fewer are working on the PR side.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#68
The dude is selling an IDE.

Also it's missing the point of a software factory concept

A software factory will not work infinitely forever for everything. A software factory isn't a solve anything button (aka god).

In a conventional factory, things break and fail. Process machines get poluted. Extruders get jammed.

You still need to establish intent, define what you care about, define guardrails, and of course manage the factory.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#69
post #32

Earlier quoted context omitted.

normative specifications can help, but the thesis here is that specs that define behavior of the product or even architecture are helpful but there's MORE that can be done and even though "program design" feels too in the weeds it's still essential if you care about maintainability

Isn't maintainability mostly about applying basic engineering principles e.g. separation of concerns, single responsibility, dependency inversion, open-closed etc..? If these topics are addressed in the normative representation, and correctly translated by an LLM into code (especially by slicing up the specifications into measurable outcomes to set the intended foundation), then future change e.g. maintainability bec…

> Historically I know that the majority maintenance problems occur from slow continuous evolution of a system that it initially was never designed for. And the only way to address this was continuous system design.

yes exactly - this is what I'm advocating for - that you can't skip the system design, and that actually good system design goes down to the typedefs and object graph at the code level, not just mermaid charts and db schemas and service contracts.

I will highlight what a few others have said along the lines of "a sufficiently detailed spec IS code" - that is, to make the spec guaranteed to produce the code you want, the spec will look a lot like code (and will be roughly the same effort to review as the code itself anyway, saving you no time)

what I'm proposing is "how can you maximized the odds that the code WILL be good or close enough to good that its easy to get there, with the LEAST amount of human effort/attention" - how can you move fast without skipping what matters

https://haskellforall.com/2026/03/a-sufficiently-detailed-sp...

Re: Why Software Factories Fail (or: harness engineering is not enough)

#70
post #34

I think there is a fundamental issue here of what building software even means If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "W…

The machine gives you what you ask for even when that thing doesn’t exist yet.

Rather than “lights off,” utilizing information theory, decision-making theory and creativity theory makes me better at asking for the right things.

Memory is not transcribed to weights like when humans sleep. Memory is notes handed to someone on groundhog’s day who can’t remember yesterday. We hope they believe us. Don’t be too surprised when a highly entropic system introduces entropy to a project over time.

Post reply on HN