[dead]
Why Software Factories Fail (or: harness engineering is not enough)
181–190 of 296 posts
Re: Why Software Factories Fail (or: harness engineering is not enough)
#182I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…
If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. You can even take a stand that you’ll never look at the code and just ship. This is a perfect equilibrium for a Software factory…
> few users, no revenue or minimal revenue
The 900 pound gorilla has for a while been "where is all the output of these LLMs?", and this seems to be the answer; the number of apps on the various app stores has skyrocketed over the past two quarters but installs overall are actually down a little bit. So it seems like we've had an explosion of software that one or two people use. Which is a cool situation to think about but is pretty bearish for the "AI as a profitable business" idea.
Re: Why Software Factories Fail (or: harness engineering is not enough)
#183I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…
If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. You can even take a stand that you’ll never look at the code and just ship. This is a perfect equilibrium for a Software factory…
The perfect poster boy is system scripts. They’re not even published usually, but being able to just generate < 500 line scripts for everything is a huge boon.
Re: Why Software Factories Fail (or: harness engineering is not enough)
#184To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…
PR review sucks because engineers often don't optimise their code authoring for it. When they do it's a joy. PR review of agent generated code sucks because agents are, in my experience, very poor at authoring for review. This makes sense because the review process is not represented in the resulting code artifact which is what they're trained on. Agents are always producing much larger changes than should be in a si…
One thing that really bothers me about agents is they generate excessively long and detailed comments.
This sounds good for review. Except… the comments are never intent or trade-offs, they’re always the “what”, which the code then duplicates right after. Comments should be short, concise “why”. Often I look at code and think “hm this is odd, I wouldn’t have done it this way, is there some constraint I don’t know about?”
The comment should explain that, not what the code is doing. If I wanted to know what the code is doing I’d read the code!
Re: Why Software Factories Fail (or: harness engineering is not enough)
#185Earlier quoted context omitted.
Agents are also controllable to a degree. How agents author PRs is now an engineering problem.
But this is my point, they're not really controllable in this way. There are no fixed rules (like tests or linters) for small PRs, it's not good enough to do it by line count. A good PR tells a story to the reader that leaves them confident in the result. That means knowing the reader and how they think, and that is something LLMs are very bad at.
And you can't think of a way to bring an engineers mindset to solve that?
If you leave it up to raw LLM outputs you're not going to get the results you're after. If you put on your engineering hat and think about your problem statement and what tools you can bring to bear on it you'll find you can likely craft a PR authoring skill and a set of evals for that skill that satisfies your requirements with a high enough degree of reliability.
Re: Why Software Factories Fail (or: harness engineering is not enough)
#186Earlier quoted context omitted.
If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. You can even take a stand that you’ll never look at the code and just ship. This is a perfect equilibrium for a Software factory…
> few users, no revenue or minimal revenue The 900 pound gorilla has for a while been "where is all the output of these LLMs?", and this seems to be the answer; the number of apps on the various app stores has skyrocketed over the past two quarters but installs overall are actually down a little bit. So it seems like we've had an explosion of software that one or two people use. Which is a cool situation to think abo…
Marketing was a problem with apps even before AI, which, with today’s slopforests makes it even harder for consumers to see any of your trees even if they wanted to.
Re: Why Software Factories Fail (or: harness engineering is not enough)
#187Earlier quoted context omitted.
If you’re working on an app or software that has few users, no revenue or minimal revenue, tolerance to bugs is higher and just another Claude prompt away, Software factories are a perfect fit. Most personal software or hobby software or 0-1 yet-to-find-PMF startups belong in this category. You can even take a stand that you’ll never look at the code and just ship. This is a perfect equilibrium for a Software factory…
The perfect poster boy is system scripts. They’re not even published usually, but being able to just generate < 500 line scripts for everything is a huge boon.
I agree this rocks I do this almost daily
Re: Why Software Factories Fail (or: harness engineering is not enough)
#188To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…
Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…
> We should be reviewing the actual working software. Systems that make it easy and instant to demo any proposed change are what is needed.
"Program testing can be used to show the presence of bugs, but never to show their absence!" -- Edsger W. Dijkstra
Re: Why Software Factories Fail (or: harness engineering is not enough)
#189Earlier quoted context omitted.
> few users, no revenue or minimal revenue The 900 pound gorilla has for a while been "where is all the output of these LLMs?", and this seems to be the answer; the number of apps on the various app stores has skyrocketed over the past two quarters but installs overall are actually down a little bit. So it seems like we've had an explosion of software that one or two people use. Which is a cool situation to think abo…
Marketing was a problem with apps even before AI, which, with today’s slopforests makes it even harder for consumers to see any of your trees even if they wanted to.
Right but if we're going to a small-batch bespoke development model then "marketing" more or less disappears because why would I pay someone to talk to an LLM when I can also talk to an LLM? The money goes from the end user to the lab in this model.
Re: Why Software Factories Fail (or: harness engineering is not enough)
#190[flagged]