Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

131–140 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#131

Post by TacticalCoder »

firasd wrote: post #34

I think there is a fundamental issue here of what building software even means If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "W…

> "Wait.. the API is already returning the data we need"

To me that's a huge one. I witnessed this firsthand more than once and the fact that you make that comment too just shows there's a problem: I've seen top frontier models go wild, adding shitload of code to spout something that, eventually (after burning shitload of tokens), worked.

While it was basically a one-line change because the data was already there on the client-side.

The thing thought it noticed a pattern and went ballistic, mimicking some existing client-server communication, to "solve" my problem.

The worst is: all vibe-coded projects are full of such nonsense and the "prompt engineers" (such engineers, much wow btw!) who wrote their magical incantations have zero clue about it.

I use these tools daily, I love the time they save me. But sheesh...

Re: Why Software Factories Fail (or: harness engineering is not enough)

#132

Post by rom1504 »

satvikpendem wrote: post #127
rom1504 wrote: post #118

Earlier quoted context omitted.

I think the simple answer is LLM cannot do long term planning. Maintainability means thinking what will happen to this code in the next year and possibly hundreds of changes and based on that selecting the right abstractions that will work in the long term. LLMs currently cannot do long term planning and specifically cannot pick abstractions that will work in the long term. I think that's the important problem to sol…

Why can they not do long term planning? They know what good software design looks like and can code accordingly. Using Fable for a greenfield project for example, it doesn't dump everything into one file but instead does indeed split out the work into logical units as needed.

Why is a very good question indeed! My guess is we have not figured out how to include long term enough tasks into the RL training.

As to whether they can today, it's pretty easy to figure out that they can't by observing the difference between asking for very high level things like "please make me a website that will be very successful" vs sending it a detailed plans on the goals and many of the specifics and then holding it accountable towards the planning.

Doing appropriate project structure and trying to optimize code quality are definitely good methods to try and increase the chance that the project will succeed long term but I would say they are only methods towards the long term goal and the LLM need to be thinking of the long term goal and how to plan towards that to achieve a much wider and subtile range of methods to achieve long term goals.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#133

Post by cadamsdotcom »

Love the idea of RL for codebase health.

And a benchmark to measure against!

Imagine a "MaintainabilityBench" that rewards models which detect code duplication while working on a task and perform some refactor instead of glibly duplicating; or that detect the need for a new architectural layer, or that hoist a type constraint so there's no need for dumb casts. You can keep on imagining scenarios.

There are probably a few hundred distinct elements to RL for. The books "Working With Legacy Code" and "Architecture of Open Source Applications" would be great fodder.

Sadly don't have time to build it, there's this mountain of reviews in front of me...

Re: Why Software Factories Fail (or: harness engineering is not enough)

#134

Post by dominotw »

cadamsdotcom wrote: post #133

Love the idea of RL for codebase health. And a benchmark to measure against! Imagine a "MaintainabilityBench" that rewards models which detect code duplication while working on a task and perform some refactor instead of glibly duplicating; or that detect the need for a new architectural layer, or that hoist a type constraint so there's no need for dumb casts. You can keep on imagining scenarios. There are probably a…

> perform some refactor instead of glibly duplicating;

wouldnt that be part of original RL though. why would it be a seperate thing.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#135

Post by cadamsdotcom »

dominotw wrote: post #134
cadamsdotcom wrote: post #133

Love the idea of RL for codebase health. And a benchmark to measure against! Imagine a "MaintainabilityBench" that rewards models which detect code duplication while working on a task and perform some refactor instead of glibly duplicating; or that detect the need for a new architectural layer, or that hoist a type constraint so there's no need for dumb casts. You can keep on imagining scenarios. There are probably a…

> perform some refactor instead of glibly duplicating; wouldnt that be part of original RL though. why would it be a seperate thing.

Because writing code that passes a test is different to passing the test while also noticing and performing a refactor.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#136

Post by orsenthil »

I find it amusing that people who are talking about Dark Software Factories, are talking about productivity in terms of number of pull requests or commits as a unit.

If we are going in the Dark Software Factory route, why aren't we calling the code units as bos (bunch of shit) yet.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#137

Post by danpalmer »

rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

PR review sucks because engineers often don't optimise their code authoring for it. When they do it's a joy.

PR review of agent generated code sucks because agents are, in my experience, very poor at authoring for review. This makes sense because the review process is not represented in the resulting code artifact which is what they're trained on. Agents are always producing much larger changes than should be in a single step, and often touch unrelated code with poor judgement of whether it should be included or not.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#138

Post by _pdp_ »

I have mixed feelings about software factories!

On one hand, our core product is just simply not fit for them at its scale. We've tried but the project is large enough to require human input for every change. But we have AI automations for light code refactoring, writing tests, UI changes etc. and they work.

On the other hand, I started a number of small experiments to see how far software factories can be pushed and while the code produced so far is nothing spectacular I could easily imagine how this can be extended in the near future. Perhaps if you start from the ground up with the idea that the code will be written that way then you can come up with strategies and architectures that accommodate it. At least this is my thinking right now.

Anyway, it is all open source and documented here https://relentless.works/ I am not sure for long I will keep this running. I provide zero direction to where this is going. I have no idea what it will happen next. It is a fun experiment.

I have another such experiment with a trading agent. I thought it will loose all of the money in short time. For a while it was stuck with no open positions after it lost a bit. I decided not to intervene and just observe the behaviour. Recently it opened new positions which was an interesting development. It is still loosing money (~ -3%) but it has not lost all of them and given the current market circumstances I would say this ain't bad at all. It just shows that perhaps we might be a bit impatient when it comes to AI.

So I think it is probably possible to build software factories but we need new concepts and a bit of change of mindset.

I hope this helps.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#139

Post by abernard1 »

gabrieledarrigo wrote: post #96
4lx87 wrote: post #18

Earlier quoted context omitted.

Gating integration behind code review is futile. I (and many other engineers) already automated it. My agent responds to review requests and reviews as me. Company policies enforcing human code review are futile. I think all these platforms chasing code review are doomed. My LLM doesn't need any of this tooling. We should be reviewing the actual working software. Systems that make it easy and instant to demo any prop…

> Code (and specs) are going to fade into obscurity I'm always skeptical when I read absolute statements like this. Especially about specs. They drive and document how the software should work, behave, and under which conditions. In the same moment you write a prompt in natural language, you are writing a spec. Why should they disappear? For the same reasoning, we could no longer need books or manuals or documents (i…

Agreed. Specs are more important than ever. Constraints: types, contracts, protocols, schemas, APIs and such are what ground the boundaries of systems.

Without them, code drift is a guarantee.

Does anyone trust that a set of human sentences typed into a box is more reliable and precise than objectively exact descriptions of data?

This is not to diminish the utility of these code factories. But surely, if anything provides resistance to drift, it's codebase consistency, standards, and things which must objectively be true for the software to function.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#140

Post by latentsea »

danpalmer wrote: post #137
rglynn wrote: post #8

To me, the thing that stands out about the whole state we're in here is PR review. Yes, in an ideal world, PRs read well, are a joy to review, reflect what you discussed etc etc. We have to be real; there is only so much we can do to that end. I'm not sure how the best teams do PR review, from my perspective it sucks . I'm talking specifically about the UX. I've always hated Github's PR page, so I typically reviewed…

PR review sucks because engineers often don't optimise their code authoring for it. When they do it's a joy. PR review of agent generated code sucks because agents are, in my experience, very poor at authoring for review. This makes sense because the review process is not represented in the resulting code artifact which is what they're trained on. Agents are always producing much larger changes than should be in a si…

Agents are also controllable to a degree. How agents author PRs is now an engineering problem.
Post reply on HN