Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

261–270 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#261
post #160

If you built a real factory, you’d basically never want it to be dark. You’d want a culture of getting wrenches out to inspect the cars being built. You’d want to care about small details. Not because we need to build cars by hand. But because looking at the real product (cars, code, etc) is the best way to make the factory better. It’s the best way to know what problems aren’t being measured, what processes need to…

> If you built a real factory, you’d basically never want it to be dark This statement seems to be factually untrue as the fully automated factories in China are indeed not illuminated for most of the time In USA/Europe we just don't have that level of automation

The “fully automated” factories have humans involved in QA, maintenance and engineering.

They often have to stop the factory and investigate and troubleshoot.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#262
post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

PR review sucks because engineers often don't optimise their code authoring for it. When they do it's a joy. PR review of agent generated code sucks because agents are, in my experience, very poor at authoring for review. This makes sense because the review process is not represented in the resulting code artifact which is what they're trained on. Agents are always producing much larger changes than should be in a si…

[flagged]

Re: Why Software Factories Fail (or: harness engineering is not enough)

#263
post #247

Earlier quoted context omitted.

It's just a long ad for his product - humanlayer.

> NO EVIDENCE AT ALL It's funny how, when the hype is strong enough, the burden of proof around the need for EVIDENCE suddenly shifts. Normally, the burden of proof is on $NEWFANGLED_THING to prove it's better than $TRIED_AND_TESTED. Software dark factories where no one looks at code are that unproven newfangled thing and all he's really saying is that, in his experience/assessment, those don't work, so he's trying t…

It's just he's qualified himself as an unreliable witness. This is not someone who deeply understands and thinks about what he does or what he advises.

The things that I know about, that he writes about, where I know he's got it a bit wrong.... I find myself questioning if he's right or not. Because he's a persuasive writer. But winning arguments doesn't make you right. This isn't a supported, balanced discussion it's a self aggrandising mismash of commonplace ideas and issues.

He's learned how to grab the mic, he believes he has the right to the mic, people are listening to him.. all three of these are wrong and until we call this out and put a stop to people like this shooting their mouths off damage is going to carry on being done.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#264
post #254

I've been building and running my software factory for 8 months now. Granted, there's no automated pulling down tasks and pushing PR right now (soon!). But after specifying what I want, it mostly goes to shipping on its own. On occasion, it does raise issues that I have to make a decision on. After doing systems evals on review, I've stopped looking at code during review for 4 months now. I do spend a lot of time up…

I'd be interested in seeing more of your setup, eg if you published a long blog post and/or repo.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#265

Earlier quoted context omitted.

Or fire the PM?

PM = politics maestro They will out-corporate you. Better to be friends with them, or failing that, move from under them

Or fuck them, fuck the managers, fuck the c-suites (intentionally decapitalized), go for a flat organizational structure like Valve where you have no one to report to (well maybe Gabe Newell or the team leader for that project occasionally), but the culture is that everyone can manage themselves organically rather than having hierarchies and forced, useless reportings

Re: Why Software Factories Fail (or: harness engineering is not enough)

#267
post #254

I've been building and running my software factory for 8 months now. Granted, there's no automated pulling down tasks and pushing PR right now (soon!). But after specifying what I want, it mostly goes to shipping on its own. On occasion, it does raise issues that I have to make a decision on. After doing systems evals on review, I've stopped looking at code during review for 4 months now. I do spend a lot of time up…

I’m most curious about what you’ve built with the code factory. Is it just building itself or do you have any outputs that validate it for building something other than AI coding tools?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#268

Earlier quoted context omitted.

PM = politics maestro They will out-corporate you. Better to be friends with them, or failing that, move from under them

Or fuck them, fuck the managers, fuck the c-suites (intentionally decapitalized), go for a flat organizational structure like Valve where you have no one to report to (well maybe Gabe Newell or the team leader for that project occasionally), but the culture is that everyone can manage themselves organically rather than having hierarchies and forced, useless reportings

Valve ships almost nothing, and the engineering quality is mediocre. Their game design is good, but subsidized by a unique early mover monopoly market owner position.

There is almost nothing that can be generalized, and people who hold up their organization as an example to strive for are utterly clueless

Re: Why Software Factories Fail (or: harness engineering is not enough)

#269
post #254

I've been building and running my software factory for 8 months now. Granted, there's no automated pulling down tasks and pushing PR right now (soon!). But after specifying what I want, it mostly goes to shipping on its own. On occasion, it does raise issues that I have to make a decision on. After doing systems evals on review, I've stopped looking at code during review for 4 months now. I do spend a lot of time up…

This has been my experience as well. While not a full "factory", I've got several mature vibed software projects.

More often than not you can "feel" when the quality has degraded. Many of these pieces leave you hanging on the assumption that once it's slopped up that's the game, it's over, you're cooked, straight to jail.

But I've yet to reach that position and be unable to pop the hood, identify the issue, and agentic-ly clean it up.

The post explicitly says it's not a skill issue, but I've yet to see any situation where an average engineer can't unslop their codebase.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#270

Earlier quoted context omitted.

I'm not sure how the best teams do PR review.. They don't do them. They discuss the changes as the team are doing them (software design and architecture), they automate all the things that would be nitpicks (linting, formatting, etc), and they adhere to a strict 'Do not break the build' rule so merging in something can't be disastrous (with a lot of automated checks and tests to prove that's true). They also make sur…

> Once you do all that the PR process is pointless manual PR reviews can catch things that llms currently miss. Examples are duplicated code, lack of unit tests or introducing security issues. None of these really break the build. So just requiring "do not break the build" is a very low barrier. Tests also are useless unless you have a smart system that runs the new test WITHOUT the changes and see it break. Most tea…

manual PR reviews can catch things that llms currently miss

I didn't say LLMs though. I'm talking about the sort of code quality tooling that's been in use for decades - Sonarqube, Codacy, CodeClimate, etc for code quality, eslint for AST-based code rule checking (easily picks up duplication for example), Istanbul for code coverage, Wiz or Github Advanced Security for security.

These are well known industry standards for knowing a code change is correct without giving it to an LLM to guess at. If you have them, and they're well configured, and you maintain the tooling properly, then PRs become unhelpful gates that don't find problems.

Post reply on HN