Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

251–260 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#251

Post by sathish316 »

high_na_euv wrote: post #236
sathish316 wrote: post #149

I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…

>If coding is only math and there is only one way to translate a requirement into an implementation, t What?? It is definitely not true

I’m of course saying there are infinite ways to implement a non-trivial feature.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#252

Post by fatata123 »

copperx wrote: post #211
canpan wrote: post #152

Earlier quoted context omitted.

It's even worse: Customer says I want X! One or more of the following can be true: - Customer really wants X - Customer doesnt actually need X at all - Customer wants Y not X but couldnt express himself - Customer B and C do NOT want X at all - Customer is just messing with you for no reason

Why deliver one solution? You're thinking small. If LLMs were fast enough, they could build A,B,C,...,Z in parallel. So that's what will end up happening. The operator will just have to make sure there's no combinatorial explosion of the possible requirements.

[dead]

Re: Why Software Factories Fail (or: harness engineering is not enough)

#253

Post by mrbnprck »

boredtofears wrote: post #105
mrbnprck wrote: post #25

I've recently started experimenting with grounding LLM driven implementation/verification on RFC based normative specifications, to avoid having to manually steer the LLM during implementation and dealing with reviewing sloppy pull requests. It works quite well, as it puts your entire focus on writing (hopefully) unambiguous specifications vs. having to discuss unwanted changes with an LLM during code-review. One fla…

I'm curious how that works - are you actually hand writing specs with RFC level detail yourself completely or is it more of an LLM assisted effort? It seems like writing a comprehensive spec manually would almost be as much (if not more) work than the implementation.

I'm writing the specs by hand.

The LLM solely helps with research and retaining the normative document layout (RFC 7322).

Since the spec grounds the underlying implementation I have to fully understand it, writing the spec manually prevents infecting the spec with the typical LLM specification slop.

When writing the specs, the goal is to systematically keep the right balance between scope and depth:

- Scope takes the most effort because it's the only thing preventing the model from escaping an intended boundary, that includes things like stable constraints/interfaces/data models

- Depth is something where I can tolerate "non-normative" interpretations by the LLM. Unless I need a stable implementation, I can decide how much I manually invest into depth vs. letting the model figure the "good enough" solution.

Since the scope/boundaries are explicitly set, the internals are pretty encapsulated and the model comparing an implementation with the spec can identify the "non-normative" pieces and communicate those to me. Then I can decide between "good enough" or to refine these non-normative implementations and make them normative.

From my various experiments the spec vs. code ratio is roughly 1/8.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#254

Post by iamwil »

I've been building and running my software factory for 8 months now. Granted, there's no automated pulling down tasks and pushing PR right now (soon!). But after specifying what I want, it mostly goes to shipping on its own. On occasion, it does raise issues that I have to make a decision on. After doing systems evals on review, I've stopped looking at code during review for 4 months now.

I do spend a lot of time up front specifying what I want. My prompts aren't one-liners, but rather an interview process where we work through all the open questions and ambiguity.

I haven't hit the wall that the OP talked about (when agents just can't seem to make the right changes, and it's impossible for me to go in and change things manually). I used a lot of guardrails such as plan reviews, browser-based QA, adversarial reviews, unit tests, linters, typecheckers, post-commit hooks, and formal method traces. I also specified engineering principles that steers the code base to minimize state and side-effects: functional core; imperative shell, make impossible states impossible, use pure functional style, etc.

There are times, when I can feel a part of the code base is messy without looking at it, because the agent will make recurring mistakes in the same part of the code base over time. What I found the agent was doing over time is that it's been layering state variables as requirements were discovered. So what helps is to ask it to refactor all these state variables into a single sum type. And if the state machine for it is complicated, I'll ask it to write a formal model of the state in Quint. Then I'll generate traces that get run as unit tests, and ask it to write the code against that.

So while the code base isn't exactly Brownfield, it's over a year old now. As for the code base, there's a backend and a frontend. I think it helps that I established a clear pattern I wanted. You code are like memes: agents will just copy patterns they see in the code base. When it does have to create a new part of the system, I found Sonnet-level models tend to draw system boundaries in all the wrong places. Opus is better. I don't yet know about Fable.

Happy to answer any questions about my workflow.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#255

Post by skydhash »

monkpit wrote: post #176
dhorthy wrote: post #128

Earlier quoted context omitted.

i guess to clarify my contention: generic prompting like "review this code" or "make the architecture better" will raise the floor but cannot come close to human-quality code without humans understanding what code exists and what to ask for. Obviously cursor and other labs have a stake in this going one direction, but I don't buy it, and I don't think you should either. In fact, "Rebuild sqlite from spec" has all the…

And what percentage of the software in the world needs to come close to human-quality code? I’d argue the percentage of the whole is VERY low.

I’d say all of them. Most people use computers as tools or as entertainment platforms. While a bug may not be life threatening in most cases, it is annoying and stress generating.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#257

Post by RugnirViking »

shermantanktop wrote: post #208
podocarp wrote: post #199

Earlier quoted context omitted.

Also quite common for me is like someone says X is good (usually a PM) but common sense says it's crap, and when you actually ask the user they also say it's crap. If only LLMs could email a bunch of people for second opinions.

Or fire the PM?

PM = politics maestro

They will out-corporate you. Better to be friends with them, or failing that, move from under them

Re: Why Software Factories Fail (or: harness engineering is not enough)

#259

Post by antonvs »

> We haven't even hit AI yet, and there are already several loops in this picture.

What a surprise! And here I was thinking that loops had only just been invented, to use in agent harnesses, as described in the famous paper “Loops Are All You Need.”

Re: Why Software Factories Fail (or: harness engineering is not enough)

#260

Post by softwaredoug »

raincole wrote: post #162
softwaredoug wrote: post #157

If you built a real factory, you’d basically never want it to be dark. You’d want a culture of getting wrenches out to inspect the cars being built. You’d want to care about small details. Not because we need to build cars by hand. But because looking at the real product (cars, code, etc) is the best way to make the factory better. It’s the best way to know what problems aren’t being measured, what processes need to…

https://en.wikipedia.org/wiki/Lights_out_(manufacturing) > Xiaomi has an 860,000 square foot factory in Changping, Beijing capable of manufacturing 10 million smartphones a year across 11 fully automated production lines. > ASE Group, a Taiwanese company that handles outsourced semiconductor packaging and testing have 56 lights out factories. It has always been the goal to make factories go dark.

There are over 700 engineers that keep the ASE factories going

> By 2024, the company has established 56 lights-out factories, trained more than 700 automation engineers,

https://www.aseglobal.com/csr/sustainability-governance/smar...

Post reply on HN