Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

91–100 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#91

Post by jadar »

jdlshore wrote: post #82
jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

I think it’s because maintainability is a factor of software design, and good design is squishy and hard to articulate. There’s no good metrics for design. All the tools that purport to measure it are doing trivial things like cyclomatic complexity, and can only tell you when things have gotten really really bad, but not distinguish between “good” and “pretty bad.” Design skills take about 10 years to develop, in my…

> It’s just not stuff LLMs are good at.

But then you're back at the question from the OP of "why not?" Is it lack of good examples in the training data? Is there something of code "goodness" that goes beyond phenomenon that are measurable? Is it that LLMs lack in the ability to organize and think abstractly?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#92

Post by lozenge »

xorcist wrote: post #12
rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

> group file changes together based on theme, add some commentary Isn't that what commits are? Or ... should be?

No, commits always display the files in a fixed order, and then display changes from line 1 to line N. An AI could select the order to display in, add per hunk commentary and automatically adjust how much context lines are displayed.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#93

Post by dhorthy »

molsongolden wrote: post #87
syndacks wrote: post #6

Dex you aren't part of the slop cannon, you _are_ the slop cannon

Just in case this isn't a compliment, I'll note that I think Dex has been pretty good about admitting when they were wrong, explaining why they were wrong, and what they're doing now instead.

we out here trying

Re: Why Software Factories Fail (or: harness engineering is not enough)

#94

Post by rapatel0 »

dhorthy wrote: post #75
rapatel0 wrote: post #68

The dude is selling an IDE. Also it's missing the point of a software factory concept A software factory will not work infinitely forever for everything. A software factory isn't a solve anything button (aka god). In a conventional factory, things break and fail. Process machines get poluted. Extruders get jammed. You still need to establish intent, define what you care about, define guardrails, and of course manage…

and build it incrementally! You don't have to build the entire software factory at once. You don't have to mastermind the whole future system, instead you're actually stacking and layering these small, isolated problems. I think that's a really good approach to start getting value tomorrow or this week without saying, "I'm going to revolutionize how we ship." It's just: 1. Find places where you can use agents. 2. Fig…

Totally. Also to add to that. I think (like any factory) you need to build in systems to allow you to have observe and service the machines make process changes as needed.

You're the foreman/factory manager. You can optimize the factory over time upgrade machines, place machines closer or add conveyor belts for more productivity.

The only think that slightly worries me is that the Labs are clearly incorporating the best in class logic from their datasets.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#95

Post by layer8 »

4lx87 wrote: post #18
rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

> We should be reviewing the actual working software. Systems that make it easy and instant to demo any proposed change are what is needed.

You’re arguing for experiment over logical proof or reasoning. That’s qualitatively a very different thing, and the former is no adequate replacement for the latter.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#96

Post by gabrieledarrigo »

4lx87 wrote: post #18
rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

> Code (and specs) are going to fade into obscurity

I'm always skeptical when I read absolute statements like this.

Especially about specs. They drive and document how the software should work, behave, and under which conditions. In the same moment you write a prompt in natural language, you are writing a spec.

Why should they disappear?

For the same reasoning, we could no longer need books or manuals or documents (is it valuable to keep around that doc about F-22 specs?).

Re: Why Software Factories Fail (or: harness engineering is not enough)

#97

Post by jdzikowski »

roncesvalles wrote: post #78
tripleee wrote: post #20

Earlier quoted context omitted.

What's your job as an "engineer" in this post-automated world? QA?

The role of a software engineer is foremost about the ownership of software systems, and secondarily about modifying them or developing new ones. Software development is a continuous operational undertaking, like having a garden, and not a one-shot "task" like designing a logo. Systems need to be tended to and somebody needs to know how they work, especially if they're business-critical or (in the case of software co…

> The role of a software engineer is foremost about the ownership of software systems, and secondarily about modifying them or developing new ones.

Yes! That's sooo accurate. Tools you use change, the level of abstraction you operate changes, but the role remains.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#98

Post by DaiPlusPlus »

4lx87 wrote: post #55
dhorthy wrote: post #41

Earlier quoted context omitted.

blake smith has a really good post on this - that mental alignment among the team is the primary purpose of code review - https://blakesmith.me/2015/02/09/code-review-essentials-for-...

That is a good post. Thank you for sharing. I can't help but notice that this person is speaking as a programmer, not the person in charge of employing programmers. Is that what the business values code review for? In my experience, managers enforce code review as a quality control mechanism: the "Find Bugs" step in the article's pyramid. And LLMs are already better than human reviewers at the "Find Bugs" part. Desig…

> In my experience, managers enforce code review as a quality control mechanism: the "Find Bugs" step in the article's pyramid.

Nope.

To reduce it down to a single word, I'd say we do code-reviews to assess "taste".

-----

Code-review is one part of a larger process (the SDLC!), and while review does help with "finding bugs" it is not the singular reason why we do it; and other parts of the SDLC are concerned with finding bugs in the first place, namely the various Test and QA steps in the process; when your code is in the (peer) code-review step then there's an expectation that the author already identified and resolved actual bugs/defects (i.e. where actual-program-behaviour deviates from the spec[1]), so a PR for new functionality is expected to include unit and integration tests to demonstrate that.

Now Claude will gladly take a Jira ticket, write-up a plan/spec, write tests, implement the feature, verify the tests pass, address static-analysis issues, push branch, and submit the PR - and if-the-program-works then it's "correct" and so surely there's nothing really to review and so merging the changes should be a breeze... but I find myself rejecting these PRs all the time because these agents still "just don't get it"[2]. (But I'm sure they'll "get it" eventually; you can't stop progress). We can revisit this topic when we get there; but for now I'm going to reject an AI-authored PR that eschews it.

Good taste is important.

[1] I'm not going to pretend anyone actually writes any kind of spec (informal or otherwise) for the vast majority of software out there; but an unwritten-spec exists when you mentally combine a vague Jira ticket, platform-conventions and common-sense (and that's how Claude in an Agentic Loop works too, except it always has to write-out the Spec.md/Plan.md to disk first, whereas us humans keep things in our head).

[2] I cannot define "it".

Re: Why Software Factories Fail (or: harness engineering is not enough)

#99

Post by jeremyjh »

4lx87 wrote: post #18
rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

This entire article is about the fact that the codebase deteriorates over time when you work this way. If they are correct, it will take longer and longer to develop PRs, and more and more cycles to QA them. Eventually the code will be unsalvageable and it might get there before anyone notices a problem. I don’t know if they are right but I know codebases can get there with humans at the helm - and the humans can adapt and learn.

This is not a certain fate but it sure looks likely and is a serious risk. Some software is so low stakes that you can get away with it I’m sure - there are always trade offs.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#100

Post by layer8 »

2001zhaozhao wrote: post #16

> When I say maintainability, I mean the specific thing where it becomes really, really hard to change one part of the codebase without breaking another part. The corollary of agents being bad at maintainability but good at coding is that you can vibecode all the parts where maintainability doesn't matter. So if you build a (domain-specific) modular architecture for your software first you can then just let your soft…

Except that there is typically a feedback loop between the implementation and the module interfaces. While implementing, you discover aspect that makes you adjust, and sometimes completely alter, the interfaces and module boundaries. And that’s not just for an initial implementation, it continues to happen as new requirements come in over the lifetime of the software. Ostensible implementation details continue to inform the architecture.
Post reply on HN