Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

31–40 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#31

Post by WorldMaker »

rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

> I'm not sure how the best teams do PR review, from my perspective it sucks. I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed by pulling down the branch and opening the diff with $EDITOR.

When $EDITOR = VSCode there's a shortcut on the GitHub PR page: if you type a . it opens in github.dev in a VSCode instance.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#32

Post by dhorthy »

mrbnprck wrote: post #25

I've recently started experimenting with grounding LLM driven implementation/verification on RFC based normative specifications, to avoid having to manually steer the LLM during implementation and dealing with reviewing sloppy pull requests. It works quite well, as it puts your entire focus on writing (hopefully) unambiguous specifications vs. having to discuss unwanted changes with an LLM during code-review. One fla…

normative specifications can help, but the thesis here is that specs that define behavior of the product or even architecture are helpful but there's MORE that can be done and even though "program design" feels too in the weeds it's still essential if you care about maintainability

Re: Why Software Factories Fail (or: harness engineering is not enough)

#33

Post by jadar »

> So, why can't models do software maintainability?

I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad design?" Of course the models aren't good at it ... they're not good at anything until you've tuned them and put them in a harness that rewards good edits and throws away (improves) bad edits. That doesn't explain why "models can't do software maintainability." The real question is why harnesses can't do software maintainability, and how to build a system that can do it. (I suppose that's the purpose of the ad at the bottom of the page.)

Re: Why Software Factories Fail (or: harness engineering is not enough)

#34

Post by firasd »

I think there is a fundamental issue here of what building software even means

If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "Wait.. the API is already returning the data we need". "Wait... let's not add customers to the report who have not been active in the past year". Stuff like that

Re: Why Software Factories Fail (or: harness engineering is not enough)

#35

Post by [deleted] »

2001zhaozhao wrote: post #17
fishtoaster wrote: post #15

There's some good ideas and points in here, but this bit threw me: > # We tried this > In July 2025 we went full lights-off Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before. I feel like any perspective/experience on "what agents can/can't do" from be…

I had a bit of this impression when reading the post as well as the authors' product website. A lot of it does seem to be stuck in 2025. For instance I think their post "long-context isn't the answer" on their website post straight-up isn't accurate, and gives me the impression they are just extrapolating previous performance to new models. In my experience, Opus 4.6 and newer have worked very reliably for long conte…

[deleted]

Re: Why Software Factories Fail (or: harness engineering is not enough)

#36

Post by dhorthy »

firasd wrote: post #34

I think there is a fundamental issue here of what building software even means If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "W…

yeah I 100% agree - and I think the most popular coding agent workflows / skill kits are designed to pull those insights and intuition out of humans in a way that optimizes for the developer's experience building the plans or building the code, e.g.

- claude code plan mode - mattpocock/skills - obra/superpowers - research/plan/implement

etc etc

Re: Why Software Factories Fail (or: harness engineering is not enough)

#37

Post by Lindby »

mrbnprck wrote: post #25

I've recently started experimenting with grounding LLM driven implementation/verification on RFC based normative specifications, to avoid having to manually steer the LLM during implementation and dealing with reviewing sloppy pull requests. It works quite well, as it puts your entire focus on writing (hopefully) unambiguous specifications vs. having to discuss unwanted changes with an LLM during code-review. One fla…

But do you actually gain anything if you need to write detailed specifications? That seems just as time consuming as writing code, but less gratifying.

Code is just detailed specifications on how things should operate.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#38

Post by d_silin »

My radical opinion is that LLMs are harmful for software development - they are the ultimate "goto" operator. All actual code should be written by a human developer.

Instead, use them in adversarial mode - run QA scenarios using LLM agent as a substitute for end user to do bug discovery.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#39

Post by dhorthy »

jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

fair point, this is the thing I struggled most to extract out while writing it - if you can propose an RL environment that penalizes a model for bad design, then I'm all ears - right now there's no fast oracle/verifier for this (as stated in the post)

My current evolving take on "how would you build such a thing" is you need to tee up a roadmap of 20 features and feed them to a model one at a time, so it can't design up front for what's coming.

That way if it builds the first 10 features and the codebase goes to slop, it get's penalized when it can't build features 11-20, or when those features take wayyy more tokens/time/cycles than a model that maintains a clean codebase can do.

This is how most real software is built by most teams - incrementally, getting feedback from users along the way, and steering goals in response.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#40

Post by dhorthy »

boron1006 wrote: post #27
rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

I just dont think LLMs are very good at judging importance or summarizing code. I tried experimenting with what is ultimately a treesitter based approach - https://github.com/0x007BA7/codebook And really liked it. Definitely nowhere near production ready but I think theres room for a player to come in and do something similar.

yeah that was another thing i hoped would pour through here - that deterministic systems are much better for evaluating quality (test, linters, cyclomatic complexity, etc) - but that we don't have such a system for code maintainability, at least not one that's widely accepted or adopted
Post reply on HN