Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

51–60 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#51

Post by Fordec »

jadar wrote: post #33

> So, why can't models do software maintainability? I feel like the explanation does nothing to actually elucidate why models can't do it. Is it an inherent weakness of LLMs? Training processes? The typical "this is crap" that we constantly hear? It goes on to write about RL and how there's no penalty for bad design. But that sort of side-steps the question and makes you ask: "why not do RL and make a penalty for bad…

I have a hunch it comes down to the training data. We have spent decades as an industry talking up the new shiny thing, deliverables, frameworks, features, performance improvements, the one thing you don't read copious amounts of prose about is "how I maintained our system so that nobody noticed".

Re: Why Software Factories Fail (or: harness engineering is not enough)

#52

Post by piker »

dhorthy wrote: post #36
firasd wrote: post #34

I think there is a fundamental issue here of what building software even means If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "W…

yeah I 100% agree - and I think the most popular coding agent workflows / skill kits are designed to pull those insights and intuition out of humans in a way that optimizes for the developer's experience building the plans or building the code, e.g. - claude code plan mode - mattpocock/skills - obra/superpowers - research/plan/implement etc etc

Hard to see them though if you’re not in the weeds and your agents are stacking abstraction on abstraction to pass tests.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#53

Post by dhorthy »

piker wrote: post #52
dhorthy wrote: post #36

Earlier quoted context omitted.

yeah I 100% agree - and I think the most popular coding agent workflows / skill kits are designed to pull those insights and intuition out of humans in a way that optimizes for the developer's experience building the plans or building the code, e.g. - claude code plan mode - mattpocock/skills - obra/superpowers - research/plan/implement etc etc

Hard to see them though if you’re not in the weeds and your agents are stacking abstraction on abstraction to pass tests.

yes exactly

Re: Why Software Factories Fail (or: harness engineering is not enough)

#54

Post by ozhero »

This is a very well written article and he makes his arguments backed up by data.

We may choose to disagree but thats the point of healthy debate based on clearly expressed opinions.

Key point is I don't think this is AI slop which is way too common in long form articles these days and in keeping with the whole point of his article.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#55

Post by 4lx87 »

dhorthy wrote: post #41
20k wrote: post #22

Earlier quoted context omitted.

>I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. This is also known as being a terrible engineer. If a company enforces human review and someone deliberately tries to circumvent this with an LLM, I'd fire that person in an instant The reason for human code review is: 1. So *you* understand what's going…

blake smith has a really good post on this - that mental alignment among the team is the primary purpose of code review - https://blakesmith.me/2015/02/09/code-review-essentials-for-...

That is a good post. Thank you for sharing.

I can't help but notice that this person is speaking as a programmer, not the person in charge of employing programmers. Is that what the business values code review for? In my experience, managers enforce code review as a quality control mechanism: the "Find Bugs" step in the article's pyramid. And LLMs are already better than human reviewers at the "Find Bugs" part. Design and alignment with PMs and sales can happen before and after software delivery. At least that's how my employer sees it (and I suspect most other businesses).

It's a rough time to be a programmer who cares about the code, who understands software is a creative process, who understands that designing the software is intertwined with the code, who cares about systems. The hyper-focus on delivery was already a lot to deal with before LLMs.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#56

Post by dhorthy »

_doctor_love wrote: post #2

> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/ Seems like it went pretty well if a consulting company is now being starte…

> In order for coding with LLMs to go well, there has to be more rigor, more discipline, more good engineering hard-assedness. To reiterate, the teams seeing the best results with AI were already high-discipline and high-hygiene.

hard agree. But i don't think this is sufficient. Even formal verification has its limitations.

> AI works on data. The better the data, the better the likelihood of a desirable outcome. Code is data. If you have bad code, no matter how awesome the model you let loose on it, you can't get as good a result as if you had good code to start with. This principle has been well known in AI/ML circles since the 20th century.

hard agree. but also RL data is shaped differently than SFT data that has driven the majority of AI/ML innovations since ~2000, and its where there's so much room for innovation still. e.g. ImageNet was all just hand-labeled answer pairs.

> it's not a skill issue, it's an effort/laziness/rigor issue

I'm sorry but this feels like a semantic argument - the point of "skill issue" is "you didn't put in the effort or learn the techniques"

Re: Why Software Factories Fail (or: harness engineering is not enough)

#57

Post by arm32 »

d_silin wrote: post #38

My radical opinion is that LLMs are harmful for software development - they are the ultimate "goto" operator. All actual code should be written by a human developer. Instead, use them in adversarial mode - run QA scenarios using LLM agent as a substitute for end user to do bug discovery.

I love how we need to preface such an opinion as being "radical" nowadays.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#58

Post by dhorthy »

throwatdem12311 wrote: post #50
2001zhaozhao wrote: post #16

> When I say maintainability, I mean the specific thing where it becomes really, really hard to change one part of the codebase without breaking another part. The corollary of agents being bad at maintainability but good at coding is that you can vibecode all the parts where maintainability doesn't matter. So if you build a (domain-specific) modular architecture for your software first you can then just let your soft…

Until you inevitably need a cross cutting concern. “Oh just this one time” Then an agent sees the pattern and assumes it’s a best practice. Then your beautiful architecture is ruined. “Just this once” indeed.

this is 100% right. you have to guard the codebase patterns with your life. because the codebase is part of the prompt.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#59

Post by [deleted] »

_doctor_love wrote: post #2

> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/ Seems like it went pretty well if a consulting company is now being starte…

[deleted]

Re: Why Software Factories Fail (or: harness engineering is not enough)

#60

Post by jaytaylor »

_doctor_love wrote: post #2

> I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/ Seems like it went pretty well if a consulting company is now being starte…

Hi, I'm one of the trio from the StrongDM AI Lab.

Just a minor thing I want to clarify about the Weather Report [1] - it's framed in kind of a negative light in the article ("sparse updates"), but we've been updating it as frequently as we find a meaningful improvement in a relevant dimension. Since we launched it in February it has averaged about one update per month, as frontier labs keep racing forward!

[1] https://factory.strongdm.ai/weather-report

Post reply on HN