Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

21–30 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#21

Post by tcoff91 »

rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

PRs sucked to review long before Agents were a thing, but now it really sucks because there are more to review.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#22

Post by 20k »

4lx87 wrote: post #18
rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile.

This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant

The reason for human code review is:

1. So *you* understand what's going on, not the LLM, and people can ask you questions about it

2. Because LLMs are not that good at code review

It seems weird to brag about literally not doing your job. It sounds like you could be replaced with a python script, what value do you bring?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#24

Post by dhorthy »

fishtoaster wrote: post #15

There's some good ideas and points in here, but this bit threw me: > # We tried this > In July 2025 we went full lights-off Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before. I feel like any perspective/experience on "what agents can/can't do" from be…

my perhaps controversial take is that opus 4.1 was smarter than 4.5 for complex engineering work, but 4.5 was faster and "squishier" - it responded better to simpler prompts, it read between the lines of user input better, and that this was really important to converting new users quickly

Re: Why Software Factories Fail (or: harness engineering is not enough)

#25

Post by mrbnprck »

I've recently started experimenting with grounding LLM driven implementation/verification on RFC based normative specifications, to avoid having to manually steer the LLM during implementation and dealing with reviewing sloppy pull requests.

It works quite well, as it puts your entire focus on writing (hopefully) unambiguous specifications vs. having to discuss unwanted changes with an LLM during code-review. One flaw is that this only works great if you know exactly what you want, which is not always the case.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#26

Post by dhorthy »

2001zhaozhao wrote: post #16

> When I say maintainability, I mean the specific thing where it becomes really, really hard to change one part of the codebase without breaking another part. The corollary of agents being bad at maintainability but good at coding is that you can vibecode all the parts where maintainability doesn't matter. So if you build a (domain-specific) modular architecture for your software first you can then just let your soft…

yeah this is along the lines of what some friends of mine call "core vs. pragmatic modules" or even s/modules/codebase zones/

the idea that if you have a solid core and decoupled modules, you can have "zones" in your codebase where you allow the model to run wild and do a little slop, because you know the blast radius is contained

Re: Why Software Factories Fail (or: harness engineering is not enough)

#27

Post by boron1006 »

rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

I just dont think LLMs are very good at judging importance or summarizing code.

I tried experimenting with what is ultimately a treesitter based approach - https://github.com/0x007BA7/codebook

And really liked it. Definitely nowhere near production ready but I think theres room for a player to come in and do something similar.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#28

Post by dhorthy »

AIorNot wrote: post #9

Wait these arent “software factories” they are strung together ai rube goldburg machines Its crazy to me people write these articles and create standards like this is some kind of engineering standard with years of research and experience This is like calling these folks the experts on aviation: https://youtu.be/M9Yww9LG3gw?is=xgtA-xMpNy-09Asu Its still so early in the game for de facto standards - engineering teams…

interesting - i'd say my main goal is to put the current "agentic software factory" hype in the historical context of "we've actually been rube-goldberging software deploys for a while now"

Re: Why Software Factories Fail (or: harness engineering is not enough)

#29

Post by _doctor_love »

stellar_jay wrote: post #7
_doctor_love wrote: post #2

> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/ Seems like it went pretty well if a consulting company is now being starte…

> Prompts are simply not enough to steer a coding agent to the level of precision needed. Without deep programmatic verification - at all levels, formal verification is just one slice - the solutions the agent produces will always be just slightly (or very) out of true. I found this to be exactly right, and in my work I’ve come up with a taxonomy of constraint mechanisms which I keep in mind when guiding agents: gene…

That's an excellent writeup. Haven't gotten all the way through it yet but so far I'm with you.
Post reply on HN