Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

81–90 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#81

Post by dhorthy »

json_hero wrote: post #46
fishtoaster wrote: post #15

There's some good ideas and points in here, but this bit threw me: > # We tried this > In July 2025 we went full lights-off Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before. I feel like any perspective/experience on "what agents can/can't do" from be…

Did you even read further on? The section right after that addresses the step changes in current frontier models https://github.com/humanlayer/advanced-context-engineering-f...

it is absolutely wild to me that this keeps floating to top comment thread when the guy very clearly did not read even the first half

Re: Why Software Factories Fail (or: harness engineering is not enough)

#82

Post by jdlshore »

jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

I think it’s because maintainability is a factor of software design, and good design is squishy and hard to articulate. There’s no good metrics for design. All the tools that purport to measure it are doing trivial things like cyclomatic complexity, and can only tell you when things have gotten really really bad, but not distinguish between “good” and “pretty bad.”

Design skills take about 10 years to develop, in my experience, and a lot of smart engineers go through a phase where they make gawdawful messes in the name of “magically easy” designs around the 5 year mark.

Design requires wisdom born of experience, isn’t measurable, and requires forming a mental model of how the system hangs together as a whole. It’s just not stuff LLMs are good at.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#83

Post by jadar »

dhorthy wrote: post #39
jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

fair point, this is the thing I struggled most to extract out while writing it - if you can propose an RL environment that penalizes a model for bad design, then I'm all ears - right now there's no fast oracle/verifier for this (as stated in the post) My current evolving take on "how would you build such a thing" is you need to tee up a roadmap of 20 features and feed them to a model one at a time, so it can't design…

Isn't that how AI written software gets better too? By steering the model towards the goals of the user?

I don't know if it's a question of how many features to feed to the model, either. Of course, overwhelm the context with too many features and it will get confused. But that's where the memory management idea that G. Huntley talks about is helpful. You're trying to steer the model within its memory limits towards a certain goal.

The problem is getting it to produce "good" code. Formalizing what that means is the task of the programmer. How do you steer a model towards always, or more often, producing good code so that you don't have to do rework? That's the same problem as with a junior engineer, but the way you do it is different. Right now we're trying to do it with mountains of prompts — which sort of works but has diminishing returns — and with onerous code reviews. We've seen this get better over time, but I think some more mechanical methods will help as we figure out how best to steer the models.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#84

Post by maerF0x0 »

I'm currently learning about Cost functions and Regression in the Machine learning course on coursera[1], and I cannot help but be struck by the similarities of how gradient descent seeks to minimize the error between the model and the training data, how agents do the same (far less mathematically) to seek an acceptable solution to software problems, and how light evolutionary pressures seem to guide species towards a better fit solution to the problem of existence and reproduction. (I'm no expert on the evolutionary example so go easy on me!)

[1] - this course: https://www.coursera.org/learn/machine-learning/home/welcome

Re: Why Software Factories Fail (or: harness engineering is not enough)

#85

Post by computably »

amannm wrote: post #42
20k wrote: post #22

Earlier quoted context omitted.

>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant The reason for human code review is: 1. So *you* understand what's going…

It seems weird to get angry about a change that is happening and will continue to happen due to what the market demands from software, which I believe will be the speed/ability to solve problems, rather than its own stewardship

Sure, if you're shipping a 500MB mobile/web app with a 5s start-up, it probably won't make a difference today, or tomorrow, or 3 months later. The problem is when your codebase is a dumping ground for AI slop and a year from now nobody can make any changes.

Also, it's not just some subjective value put in human review. If a change causes a major outage and it turns out it's because 5 reviewers were all shirking the company-mandated responsibility of review, and it could trivially have been caught, I would 100% expect those people to get fired. It's not a process issue if people are intentionally circumventing the process.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#86

Post by Altern4tiveAcc »

>It's easy to be a little bummed by the core thesis here: "for now we're stuck reading the code".

>I was pretty excited for a world where we could just ask for things and let the models cook and not read the code and get beautiful production software

I feel so disconnected reading those things. Reading and writing code is what brings me joy. I'd never feel "bummed" or "stuck" with it.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#87

Post by molsongolden »

syndacks wrote: post #6

Dex you aren't part of the slop cannon, you _are_ the slop cannon

Just in case this isn't a compliment, I'll note that I think Dex has been pretty good about admitting when they were wrong, explaining why they were wrong, and what they're doing now instead.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#88

Post by Balooga »

Lindby wrote: post #37
mrbnprck wrote: post #25

I've recently started experimenting with grounding LLM driven implementation/verification on RFC based normative specifications, to avoid having to manually steer the LLM during implementation and dealing with reviewing sloppy pull requests. It works quite well, as it puts your entire focus on writing (hopefully) unambiguous specifications vs. having to discuss unwanted changes with an LLM during code-review. One fla…

But do you actually gain anything if you need to write detailed specifications? That seems just as time consuming as writing code, but less gratifying. Code is just detailed specifications on how things should operate.

> Code is just detailed specifications on how things should operate.

The spec defines how things should operate. Code defines how things operate.

The problem being that if there is only code, then any change to system behaviour is always correct (including bugs). Because that's what the code says.

Relying on tests won't help without a spec to create tests from. So tests are generated from code. So if a test fails, it's the fault of the test, not the code.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#89

Post by necovek »

4lx87 wrote: post #45
tripleee wrote: post #20

Earlier quoted context omitted.

What's your job as an "engineer" in this post-automated world? QA?

Producing features and fixing bugs, same as it was before. The organizational process of software development has not changed much with AI: execs decide direction and initiatives, PMs decide what to build, which is broken down into features and bug fixes that SWEs produce. In my experience organizations don't actually care how SWEs produce features, except insofar as it relates to how many and how fast the features c…

Before LLMs, "organizations" have seen code review as multiple things:

  * Share knowledge about a particular area of the codebase between multiple people
  * Share overall engineering knowledge between the parties in the review
  * Ensure maintainability of the codebase long-term
  * Ensure readability of the code
  * Catch architectural/directional omissions (yes, from the planning/architecture phase) before it's really too late and non-reversible change goes in (eg. large destructive DB schema change)
  * Ensure changes are small, self-contained, and as often as possible, reversible
  * Do some basic manual QA
  * Do comprehensive integration testing with a fully built-out system
  ...
  * And yes, catch bugs before they hit production.
A lot of the above could be fun and engaging, and especially knowledge sharing and ensuring maintainability/readability was a very motivating thing for me as a more experienced engineer having learned so much from getting good reviews when I was less experienced.

Programmers care about style to ensure readability and thus maintainability of the code, but also to keep changes minimal — if every diff converted between tabs and spaces in the entire codebase, it'd be impossible to see what has really changed with the simple tooling we generally use (one could build diffing tools that ignore changes like these, and they even exist, but are not ubiquitous).

Re: Why Software Factories Fail (or: harness engineering is not enough)

#90

Post by hansvm »

2001zhaozhao wrote: post #17
fishtoaster wrote: post #15

There's some good ideas and points in here, but this bit threw me: > # We tried this > In July 2025 we went full lights-off Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before. I feel like any perspective/experience on "what agents can/can't do" from be…

I had a bit of this impression when reading the post as well as the authors' product website. A lot of it does seem to be stuck in 2025. For instance I think their post "long-context isn't the answer" on their website post straight-up isn't accurate, and gives me the impression they are just extrapolating previous performance to new models. In my experience, Opus 4.6 and newer have worked very reliably for long conte…

FWIW, I notice the intelligence drop drastically with long contexts with Opus 4.6. It's barely usable for anything intricate and long. That long window is good for _something_, but it's not as good as a short window.
Post reply on HN