Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody has built a software factory

medium.com

Post reply on HN
Print view View original

61–70 of 81 posts

Re: Nobody has built a software factory

#61

Post by nonameiguess »

The idea of a software factory long predates LLMs or any other form of automated full-program generation. The idea was to take full CI/CD pipelines that allow for rapid deployment of new components into a running system a la popular SaaS providers and web platforms that are usually single vendor or even single application, and extend the idea to a centralized production facility used by many different application vendors and consumer endpoints. Kessel Run in the US Air Force is the largest widely-touted success story I'm aware of, but I'm sure there must have been others.

Think of Netflix and Google sharing the same developer VDIs, build tooling, unit test framework, E2E test framework, promotion gates, and possibly even a common runtime platform. It's not something the commercial world tends to think much about, but in the military world you have many different vendor collaborating on a single C2 or comms system, but individually building their own production lines for each component. Centralize that and you can reduce procurement costs by de-duplicating effort, at least in theory.

Plenty of these have been built. I can buy that no one has built a fully-automated set of agents running on common compute, storage, and network substrates that can create, verify, and ship arbitrary software from vague natural language desirements alone, but that isn't the same thing and it's kind of a shame they decided to use an already in-use name for this. As many have pointed out, real physical factories don't do this, either. They build and ship the same thing with the same physical design in arbitrary quantities, which is a much easier problem to solve. Any Linux distro or mobile app store already does that.

Re: Nobody has built a software factory

#62

Post by nzoschke »

zrail wrote: post #53
nzoschke wrote: post #43

Earlier quoted context omitted.

pre-commit and pre-push hooks sound smart. I like that those will work the same for any human or any harness vs harness hooks. Then push straight to main? Or do we need GitHub to help manage the queue https://docs.github.com/en/repositories/configuring-branches...

I think that's orthogonal to testing/ci/hooks and depends on what you're building and what stage you're at. Personally I would not run anything with real users without PRs and branch protection.

My `codex` design session landed on opening GitHub PRs and labeling some with `automerge` that we're comfortable landing on main after GH also verifies the checks.

Re: Nobody has built a software factory

#64

Post by nzoschke »

dust-jacket wrote: post #28
nzoschke wrote: post #26

The referenced https://blog.exe.dev/replace-your-ci is resonating with me. I'm spending a relatively large amount of time waiting for CI to finish running, then clicking a button for an agent to fix failures, then waiting for CI to finish again. Might as well push that down into the core development loop. Do folks have advice for doing this in practice? Hooks in the harness? Pre-commit hooks? Different GitHub configu…

just tell it to watch CI and fix errors?

Yes but this can be a slow loop.

Push to GH, run CI (in GHs relatively slow / fragile runners), poll GH APIs to see status and get errors, make changes. Then repeat.

The optimization is to bring running tests / fixing errors into the core agentic development loop, be much more confident that CI will pass on final pushed changes, then automatically queue / merge the changes.

Re: Nobody has built a software factory

#65

Post by devashish86 »

mccoyb wrote: post #21

I don’t think it’s possible to build a software factory unless you truly do not care about what you’re putting out. Agent swarms, self learning, Ralph loops, execution DAGs, spending hours trying to convey my preferences into skills, yada yada — the reality is that no matter how long I spend trying to close the amortization gap between my expertise and (insert autonomous agent loop) … I always come back to see shit t…

Same experience. I've spent hours tweaking prompts, harnesses, skills and what not to have a simple (but non trivial) project completely managed by an agent.

The key constraint was keeping the quality bar high.

If I lower the quality bar and _only_ care about a working solution, this works great. And I dare not look at the implementation details if this is the outcome I care about.

But if I want a working solution _and_ have it done the right way, then I need to be in the loop at key points. Newer models, unfortunately, have made this problem worse. I will inevitably come back to some form of bad decisions, deviations or just slop. Left unattended, it snowballs into a pile of mess. At this point, I decide if I spend time fixing it or switch my mental model to not care about the quality.

The problem is worse when there's a team involved here. Fragmented tooling , prompts, harness, model causes more divergence. Especially in quality. Senior engineers become bottleneck as the quality enforcers.

I do like the revert rate metric that Posthog tracks. This can help incrementally build the tooling to a point it can be trusted and scales.

Re: Nobody has built a software factory

#66

Post by [deleted] »

layer8 wrote: post #44
sarreph wrote: post #3

I really like the premise of this article because I am intensely interested in learning about software factories. As much as "software factory" is a buzzword, it's still SOTA and it's absolutely going to happen / keep happening in the tech industry over the next few years. However, I can't read this, not because I refuse to -- I want to -- but because my the cogs in my mind just stop whirring at all of the LLM-isms,…

The factory analogy is tenuous, because real software is always bespoke, it’s not about producing gazillion copies of the same thing. Factories are about reliability and tolerances, aspects which are inherently intractable with LLMs.

[deleted]

Re: Nobody has built a software factory

#67

Post by SAI_Peregrinus »

simonw wrote: post #54
steveklabnik wrote: post #35

Earlier quoted context omitted.

writing prose and writing code are two different things I can get good code out of LLMs. I can rarely get good writing out of them. Maybe it's a skill issue, but it's not inherently a contradiction that they're better at some things than others. It's much easier to RL coding than writing.

Being dull and predictable and repeating existing patterns are characteristics of good code, and simultaneously characteristics of bad writing.

I like to delete all AI comments, and write my own. If I can't write a comment, I don't understand the code well enough to have my name on the commit. I often delete the resulting comment, because AI likes to add many more than are necessary.

Re: Nobody has built a software factory

#68

Post by arbourtrary »

OccamsMirror wrote: post #38
sarreph wrote: post #3

I really like the premise of this article because I am intensely interested in learning about software factories. As much as "software factory" is a buzzword, it's still SOTA and it's absolutely going to happen / keep happening in the tech industry over the next few years. However, I can't read this, not because I refuse to -- I want to -- but because my the cogs in my mind just stop whirring at all of the LLM-isms,…

As soon as I read this: "Everyone Bought the Motor. Ford Rebuilt the Floor." my brain recoiled and I did not want to continue reading it.

Yep that was the exact moment for me too. The premise was interesting, but I hit that spot and just registered it as not worth proceeding.

Re: Nobody has built a software factory

#69

Post by grenbys »

Mixed thoughts on this article, here's my take: There's no need to build a Software Factory. Every company already had a software factory in pre-AI – it is a socio-technical system delivering software (e.g. R&D). The input is the product ideas, the output is those ideas shipped to customers as code running in production. With AI we now have a chance to modernise the existing factory – speed it up. Code generation is solved, all the other bits of the factory are not: cheap&fast CI at scale, tech planning, code review, validation in production, incident diagnosis, etc. Naive way of "modernisation" is to give everyone a Claude Code subscription and call it done. Proper way is to figure out where the bottlenecks are and automate them with high-quality specialised AI agents. To surface the bottlenecks you need to pump in the throughput and see where it breaks – in the same way how you'd find an air leak in a tyre. The analogy with real factories is the story on how they were modernising the cotton mills when the electricity came in: retrofitting a big-ass electric engine (give everyone Claude/Codex) does not work, instead every part of the factory was modernised with smaller electric engines (specialised AI agents) – that's when the mill started to really increase its output. Humans are still there as building agents is REALLY hard, and it is not an end-state and more of a process: e.g. AI labs models are end-of-life every year and good luck just switching a model w/o re-engineering the prompts, evals, a/b tests, etc). But the emphasis really is on the SOCIO-technical system. Building those specialised AI agents is not only hard because of the non-determinism of underlying technology, but because those agents will interact with humans, create incentives, second order effects in a complex system. Every existing software factory is just a projection of a given engineering culture. Things you value as an R&D org, things you've optimised for over the years - they shape whatever the factory you have - with or w/o AI

Re: Nobody has built a software factory

#70

Post by nzoschke »

mccoyb wrote: post #21

I don’t think it’s possible to build a software factory unless you truly do not care about what you’re putting out. Agent swarms, self learning, Ralph loops, execution DAGs, spending hours trying to convey my preferences into skills, yada yada — the reality is that no matter how long I spend trying to close the amortization gap between my expertise and (insert autonomous agent loop) … I always come back to see shit t…

> I always come back to see shit that I have opinions about and need to polish

Definitely but can't the factory analogy still apply here?

We're now workers on a factory assembly line.

The assembly line still needs people to design and build machines, press buttons to operate machines, verify widgets are correct for their stage of assembly, and verify the final product is correct and ready to leave the assembly line.

Importantly it needs people that spot defects and send things back, and press the "big red button" if something is grossly wrong with the whole process.

The goal is to make this as efficient as possible working with our agents.

I don't think the agentic coding tools are anywhere close to making this awesome yet, but at the same time the best practices of "dev ops" need to be updated to involve agents.

Post reply on HN