Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

71–80 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#71

Post by claytonjy »

firasd wrote: post #34

I think there is a fundamental issue here of what building software even means If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "W…

> There are 'points of view' that emerge during coding

I know it’s a bit cliche at this point, but this harkens to “programming as theory building”[0] which I agree is easy to lose out on when embracing agentic coding today.

[0]: https://gwern.net/doc/cs/algorithm/1985-naur.pdf

Re: Why Software Factories Fail (or: harness engineering is not enough)

#72

Post by dhorthy »

claytonjy wrote: post #71
firasd wrote: post #34

I think there is a fundamental issue here of what building software even means If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "W…

> There are 'points of view' that emerge during coding I know it’s a bit cliche at this point, but this harkens to “programming as theory building”[0] which I agree is easy to lose out on when embracing agentic coding today. [0]: https://gwern.net/doc/cs/algorithm/1985-naur.pdf

I have had the phrase "programming is building a theory" spinning in my head for days, especially since watching this pragmatic engineer pod with Kent Beck https://www.youtube.com/watch?v=ddHQQtjIOpw

Re: Why Software Factories Fail (or: harness engineering is not enough)

#73

Post by zingar »

Enjoyed most of this but unconvinced by the program design part. If I see an agent writing function signatures or listing which functions to edit in a plan that tells me that I’ve given it too big a vertical slice. I always delete the code guesses. The thing that writes the code must always spend some time discovering where/what to write or it won’t have the right context. Or put another way: “decide first, act later” always feels worse than act-learn-act.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#74

Post by jaytaylor »

dhorthy wrote: post #62
jaytaylor wrote: post #60

Earlier quoted context omitted.

Hi, I'm one of the trio from the StrongDM AI Lab. Just a minor thing I want to clarify about the Weather Report [1] - it's framed in kind of a negative light in the article ("sparse updates"), but we've been updating it as frequently as we find a meaningful improvement in a relevant dimension. Since we launched it in February it has averaged about one update per month, as frontier labs keep racing forward! [1] https:…

appreciate that context! I definitely did not mean to come out and say "its definitely not working" or anything, but would love to hear from y'all a retrospective on the ~5-6 month anniversary - what was right, what did we get wrong, etc

We are working on new articles to share our latest findings, so stay tuned! Overall our outlook continues to be bullish. Almost all software problems yield to a combination of the Factory Techniques covered on the strongdm.ai website. More powerful models work even better...

Re: Why Software Factories Fail (or: harness engineering is not enough)

#75

Post by dhorthy »

rapatel0 wrote: post #68

The dude is selling an IDE. Also it's missing the point of a software factory concept A software factory will not work infinitely forever for everything. A software factory isn't a solve anything button (aka god). In a conventional factory, things break and fail. Process machines get poluted. Extruders get jammed. You still need to establish intent, define what you care about, define guardrails, and of course manage…

and build it incrementally!

You don't have to build the entire software factory at once. You don't have to mastermind the whole future system, instead you're actually stacking and layering these small, isolated problems.

I think that's a really good approach to start getting value tomorrow or this week without saying, "I'm going to revolutionize how we ship."

It's just:

1. Find places where you can use agents.

2. Figure out where the right leverage points are for humans and where the right leverage points are for agents.

3. Just start building those things and plugging them into each other.

One day you'll wake up, and 80% of all of your stuff is automated.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#76

Post by sergiotapia »

Personal anecdote: I was able to set things up in such a way that multiple non technical people at my job were able to build features into our project. One person even created our own CRM. And I'm not a turbo-genius savant.

The one downside was that it required their computers to install dev dependencies, postgres, infisical for secret management, etc.

That's the next frontier. What I'm working on now independently. I plan to open source this solution, but again I'm not a savant genius. I guarantee many people are working on the same thing it's converging. Our job as engineers is becoming more of a higher level facilitator and AI "plumbing" maintenance work. How can we get the AI to empower everybody at the company while keeping the wheels turning. That's my aim.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#77

Post by Robdel12 »

I 100% whole heartily agree with this. Anyone saying the models made a huge leap in fall 2025 / spring 2026 still aren't looking at whats going on. Just this weekend I thought I had a solid plan and evals/tests to let 5.6 sol work unsupervised over night while I slept and it made an absolute mess. Somewhere in the loop it had to make a decision and it made a wrong one. Making everything beyond that trash. The code 'worked' but it had the wrong system design and wrote the most brittle tests around its assumption, validating its own decision. It turned into a nasty feedback loop for the model. This is 5.6 sol high.

The models write good/great code. I'm very happy to never write code again but models are no where near good enough to run off on their own without a human in the loop. Models LOVE to cheat. Just peek at the tests they write.

And before you come at me, I have built 7 products in the past 1.5/2 years with agentic engineering, all with users. One of those projects is _dead_ because I let the vibe go too hard at the same time Anthropic decided to nerf both their harness and their models silently. If you care to look at the source: https://github.com/Robdel12/OrbitDock I spent a week or so and like a billion+ tokens trying to refactor and save it. It just wasn't worth it.

I wish people would be pragmatic about this. I get the dream is to let it do everything and not to be in the loop, because being in the loop is exhausting. But if you want to make whatever you're building be robust and survive more than 6 months, you have to. I don't care how good your tests, plans, skills, etc are. At some point the model will have to decide something and it'll be the wrong one. Compounding the slop from there forward.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#78

Post by roncesvalles »

tripleee wrote: post #20
4lx87 wrote: post #18

Earlier quoted context omitted.

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

What's your job as an "engineer" in this post-automated world? QA?

The role of a software engineer is foremost about the ownership of software systems, and secondarily about modifying them or developing new ones.

Software development is a continuous operational undertaking, like having a garden, and not a one-shot "task" like designing a logo. Systems need to be tended to and somebody needs to know how they work, especially if they're business-critical or (in the case of software companies), the business itself.

A corollary point is that many of the rules and policies of the business are solely embedded within the software itself. There is rarely a complete set of documents that says "this is how everything works" -- if you want to know how something works, you ask a software engineer, they take a look inside the codebase and then tell you how it works. In that sense, all software engineers are also product managers (or more accurately, product managers are just crude software engineers, but that's besides the point).

Re: Why Software Factories Fail (or: harness engineering is not enough)

#79

Post by dhorthy »

zingar wrote: post #73

Enjoyed most of this but unconvinced by the program design part. If I see an agent writing function signatures or listing which functions to edit in a plan that tells me that I’ve given it too big a vertical slice. I always delete the code guesses. The thing that writes the code must always spend some time discovering where/what to write or it won’t have the right context. Or put another way: “decide first, act later…

one thing I probably didn't mention is we do the program design having already done an in-depth codebase research, with current patterns and architecture surfaced - that actually seeds every step of the flow including even the product part -

but yes if you're working in very small slices then I think it's very feasible to skip program design and just review the code as you go, and resteer live. I do this all the time for tasks that are too big for a oneshot but on the smaller side overall.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#80

Post by dhorthy »

Robdel12 wrote: post #77

I 100% whole heartily agree with this. Anyone saying the models made a huge leap in fall 2025 / spring 2026 still aren't looking at whats going on. Just this weekend I thought I had a solid plan and evals/tests to let 5.6 sol work unsupervised over night while I slept and it made an absolute mess. Somewhere in the loop it had to make a decision and it made a wrong one. Making everything beyond that trash. The code 'w…

> I spent a week or so and like a billion+ tokens trying to refactor and save it. It just wasn't worth it.

this is exactly what happened to github.com/humanlayer/humanlayer - it was overslopped and we reset from scratch to build it right - spent 2 weeks in VS CODE - not even cursor, plumbing the core patterns from scratch. codebase is part of the prompt, yada yada

> I wish people would be pragmatic about this

that might be the tl;dr for the whole post haha

Post reply on HN