Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

211–220 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#211

Post by copperx »

canpan wrote: post #152
sathish316 wrote: post #149

I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…

It's even worse: Customer says I want X! One or more of the following can be true: - Customer really wants X - Customer doesnt actually need X at all - Customer wants Y not X but couldnt express himself - Customer B and C do NOT want X at all - Customer is just messing with you for no reason

Why deliver one solution? You're thinking small. If LLMs were fast enough, they could build A,B,C,...,Z in parallel. So that's what will end up happening. The operator will just have to make sure there's no combinatorial explosion of the possible requirements.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#212

Post by RamblingCTO »

fishtoaster wrote: post #15

There's some good ideas and points in here, but this bit threw me: > # We tried this > In July 2025 we went full lights-off Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before. I feel like any perspective/experience on "what agents can/can't do" from be…

The obvious statement would then be: 2025 we went all in, 2026 still nothing improved substantially. I also agree with that statement, none of the frontier models handle drift any better. Shotgun surgery and all that. You're waving him off by waving him off, but provide no arguments whatsoever. Is your experience substantially different?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#213

Post by RamblingCTO »

4lx87 wrote: post #55
dhorthy wrote: post #41

Earlier quoted context omitted.

blake smith has a really good post on this - that mental alignment among the team is the primary purpose of code review - https://blakesmith.me/2015/02/09/code-review-essentials-for-...

That is a good post. Thank you for sharing. I can't help but notice that this person is speaking as a programmer, not the person in charge of employing programmers. Is that what the business values code review for? In my experience, managers enforce code review as a quality control mechanism: the "Find Bugs" step in the article's pyramid. And LLMs are already better than human reviewers at the "Find Bugs" part. Desig…

I can tell you that, because I'm that person.

What I care about with my department:

- alignment on new stuff being introduced into code bases (the bigger it is, the more important to consider promixity on things the teams are working on tho). also: maybe finding a more optimal solution together, implies a proper PR process (which I always created with my teams, it takes time and we plan for that)

- code quality assessment: maintainability, regression tests, taste, security etc. etc.

- code is improved by expecting a review in the end. if you know you're not reviewed, you might rush and push out suboptimal code. hurts the baseline

Maintainability, bugs, incidents, team understanding, complexity (all impacting future delivery, uptime, user retention, onboarding etc.) are all metrics I care about as they impact the software and the company as a whole directly. A lot of CTOs or other execs/managers might not care, but there are a lot of bullshitters out there. Or maybe I'm just used to complex software in regulated/hard fields, I don't know.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#214

Post by RamblingCTO »

NateEag wrote: post #188
4lx87 wrote: post #18

Earlier quoted context omitted.

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

> We should be reviewing the actual working software. Systems that make it easy and instant to demo any proposed change are what is needed. "Program testing can be used to show the presence of bugs, but never to show their absence!" -- Edsger W. Dijkstra

And it doesn't take into account maintainability, complexity, security etc. I think the commentator above you is not a SWE. No one could be that ignorant

Re: Why Software Factories Fail (or: harness engineering is not enough)

#215

Post by weatherlite »

janalsncm wrote: post #147

Either you need to understand how your codebase works or you don’t. Claude can write the code for you but it can’t understand it for you. That part has to happen at human speeds. There are cases where you don’t have to understand everything, but I think that’s a more nuanced question. All of the above is true even if Claude writes perfect code.

> Either you need to understand how your codebase works or you don’t.

It's an interesting point. We can also think about it perhaps as a non binary thing - you need X amount of understanding in a specific codebase to be effective. Even before LLMs in large codebases no one understood it all; but we at least mostly understood our own PRs and our own areas of expertise in the codebase.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#216

Post by svara »

fishtoaster wrote: post #103
json_hero wrote: post #46

Earlier quoted context omitted.

Did you even read further on? The section right after that addresses the step changes in current frontier models https://github.com/humanlayer/advanced-context-engineering-f...

As I mentioned explicitly in the comment you're replying to: yes, I did read that section. Did you not bother to read my whole comment before telling me to read more?

OK, but having read that section you'll see how

> just writes off any improvement

Is not what that section contains?

There's a more nuanced point about capabilities improving in some ways but not others (ability to one-shot apps vs. maintenance of existing code over time).

Re: Why Software Factories Fail (or: harness engineering is not enough)

#217

Post by regularfry »

monkpit wrote: post #176
dhorthy wrote: post #128

Earlier quoted context omitted.

i guess to clarify my contention: generic prompting like "review this code" or "make the architecture better" will raise the floor but cannot come close to human-quality code without humans understanding what code exists and what to ask for. Obviously cursor and other labs have a stake in this going one direction, but I don't buy it, and I don't think you should either. In fact, "Rebuild sqlite from spec" has all the…

And what percentage of the software in the world needs to come close to human-quality code? I’d argue the percentage of the whole is VERY low.

The whole point is that if you don't maintain high quality, any software will degrade to the point where the factory breaks. "Human-quality" here isn't quite the right measure because human software factories also bog down in exactly this way.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#219

Post by NamlchakKhandro »

shermantanktop wrote: post #208
podocarp wrote: post #199

Earlier quoted context omitted.

Also quite common for me is like someone says X is good (usually a PM) but common sense says it's crap, and when you actually ask the user they also say it's crap. If only LLMs could email a bunch of people for second opinions.

Or fire the PM?

ask Ed Zitron how well that statistically pans out

Re: Why Software Factories Fail (or: harness engineering is not enough)

#220

Post by nevertoolate »

weatherlite wrote: post #215
janalsncm wrote: post #147

Either you need to understand how your codebase works or you don’t. Claude can write the code for you but it can’t understand it for you. That part has to happen at human speeds. There are cases where you don’t have to understand everything, but I think that’s a more nuanced question. All of the above is true even if Claude writes perfect code.

> Either you need to understand how your codebase works or you don’t. It's an interesting point. We can also think about it perhaps as a non binary thing - you need X amount of understanding in a specific codebase to be effective. Even before LLMs in large codebases no one understood it all; but we at least mostly understood our own PRs and our own areas of expertise in the codebase.

I have a very good mental model in my head of each and every codebase I have ever worked on. Nothing to do with my PRs or even my team’s direct responsibilities. I could always direct a thought provoking idea to challenge the existing status quo and get a reasonable nuanced answer from code owners on slack or during a water-cooler talk. I build that mental model also by reading the code. This is my job, so I know when to tear down what or how to respond to a random product idea immediately on a meeting to asses complexity.
Post reply on HN