Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

231–240 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#231

Either you need to understand how your codebase works or you don’t. Claude can write the code for you but it can’t understand it for you. That part has to happen at human speeds. There are cases where you don’t have to understand everything, but I think that’s a more nuanced question. All of the above is true even if Claude writes perfect code.

> Either you need to understand how your codebase works or you don’t. It's an interesting point. We can also think about it perhaps as a non binary thing - you need X amount of understanding in a specific codebase to be effective. Even before LLMs in large codebases no one understood it all; but we at least mostly understood our own PRs and our own areas of expertise in the codebase.

Before LLMs you either built good abstractions to make it possible to not understand large chunks of the code base or you flailed.

A lot of the time people flailed.

The one thing Ive never seen an LLM do well is shape clean, coherent abstractions. To be fair it's a rare human skill as well but in LLMs if they don't have a direct analog in their training data they flail.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#233
post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

I'm not sure how the best teams do PR review.. They don't do them. They discuss the changes as the team are doing them (software design and architecture), they automate all the things that would be nitpicks (linting, formatting, etc), and they adhere to a strict 'Do not break the build' rule so merging in something can't be disastrous (with a lot of automated checks and tests to prove that's true). They also make sur…

> Once you do all that the PR process is pointless

manual PR reviews can catch things that llms currently miss. Examples are duplicated code, lack of unit tests or introducing security issues. None of these really break the build. So just requiring "do not break the build" is a very low barrier.

Tests also are useless unless you have a smart system that runs the new test WITHOUT the changes and see it break. Most teams I see today have the LLM write a test along with the change, without a guarantee that the tests actually guard the feature.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#235
This is interesting and all, but why should we believe any of it?

1) This guy has a track record (confessed) of making shit up, yapping on about it, and pushing it on the innocent. He's done a bunch of damage with his bullshit and now wants us to pay attention again. I mean - something something off fella.

2) There is NO EVIDENCE AT ALL that his ideas are good. He's just making stuff up.

Give me a reason. Also shouldn't he be ostracised and stripped of his wealth for his previous rubbish?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#236

I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…

>If coding is only math and there is only one way to translate a requirement into an implementation, t

What??

It is definitely not true

Re: Why Software Factories Fail (or: harness engineering is not enough)

#237

Earlier quoted context omitted.

If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. You can even take a stand that you’ll never look at the code and just ship. This is a perfect equilibrium for a Software factory…

> If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. I would rather claim that for my personal/hobby software, I apply a much higher quality standard than for the software that I…

I’m always surprised when I read comments like this. I’ve come across far more situations where individual contributions take a slapdash approach and management is disengaged than management actively imposing poor quality work.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#238
post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

I'm not sure how the best teams do PR review.. They don't do them. They discuss the changes as the team are doing them (software design and architecture), they automate all the things that would be nitpicks (linting, formatting, etc), and they adhere to a strict 'Do not break the build' rule so merging in something can't be disastrous (with a lot of automated checks and tests to prove that's true). They also make sur…

> They discuss the changes as the team are doing them (software design and architecture)

I find the format of PRs to be better suited to discussion than slack/teams messages. Key developers need to actively sign off, code suggestions are much easier, and the CICD makes sure we're not wasting time discussing code that's not compliant with obvious standards.

Obviously some level of agreement is needed before creating a ticket at all, but implementation details around say creating a terraform module to share functionality between teams do need to be discussed somewhere - and it can be difficult for any individual dev to have all the low level context.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#239
post #226

Earlier quoted context omitted.

Why deliver one solution? You're thinking small. If LLMs were fast enough, they could build A,B,C,...,Z in parallel. So that's what will end up happening. The operator will just have to make sure there's no combinatorial explosion of the possible requirements.

Right. Because If I want feature A and it is available in configuration among features A through Z I will always perfectly pick the right letter of the alphabet and I will not at all be irritated by having 26 options that I have no idea about what is the difference between them.

But I think there's anyway a point there. I watched a talk once and it was humorously said that the amount of time spent discussing whether solution A is better than B is more than implementing both A and B and seeing in practice. It's funny because it's true.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#240
post #179

Earlier quoted context omitted.

If you need to actively keep your head down, you need to quit. Doesn’t matter whether ‘the problem’ is the employer, the employee, both, or neither. They’re a bad fit for each other.

> If you need to actively keep your head down, you need to quit. This really depends on whether they are paying you decently. The number of people I've met who found a nice corner in FAANG, and decided to keep their head down and collect a steady paycheque... One may find one can swallow a lot of engineering pride for a recurring six figures

And that’s not to say anything about personal situations. Family obligations, immigration requirements, etc.

Especially in the current job market, “find a new job and quit” is going to cost you so much time and effort and for many people it will still be a lateral move in terms of company quality in regards to engineering practices. It’s hard to make these kinds of deep company discoveries until you’re in the thick of it but by then it’s too late.

Personally, I’m advocating for change where I can and not taking it personally when the manager says no. I put in enough effort not to get paged in the middle of the night which is generally plenty for management. It helps that I don’t work in a super strict business environment.

There’s more to life outside of work as well so if you can silo that bit of your life and enjoy things outside of working hours and you’re content, so be it.

Post reply on HN