Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

271–280 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#271

Post by spacecadet »

It's a skill issue... governance and culture skills. Doesn't matter at this point what model you use... a thoughtful harness with appropriate levels of HITL and strong security, governance, discipline. "We" have a very successful dark factory running... no I cant tell you about it. It's trade secrets at this point. It was incredibly hard to build, roll out, and maintain.

People have no idea that 2 years ago we hit peak performance with omni models. We won competitions with GPT-4o-mini, mini... it does not matter. Context matters. Discipline matters. Observability matters. Most of all, PEOPLE matter.

All of the fools who tried this and laid people off. Fools. It's about empowering your team, making their lives easier, less frustrating, and higher impact.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#272

Post by Terretta »

hansvm wrote: post #90
2001zhaozhao wrote: post #17

Earlier quoted context omitted.

I had a bit of this impression when reading the post as well as the authors' product website. A lot of it does seem to be stuck in 2025. For instance I think their post "long-context isn't the answer" on their website post straight-up isn't accurate, and gives me the impression they are just extrapolating previous performance to new models. In my experience, Opus 4.6 and newer have worked very reliably for long conte…

FWIW, I notice the intelligence drop drastically with long contexts with Opus 4.6. It's barely usable for anything intricate and long. That long window is good for _something_, but it's not as good as a short window.

for synthesis. read the entire whatever in one gulp and boil it down or plan what to do about it.

not for steps.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#273

Post by stevefan1999 »

nfbdhdfbf wrote: post #268
stevefan1999 wrote: post #265

Earlier quoted context omitted.

Or fuck them, fuck the managers, fuck the c-suites (intentionally decapitalized), go for a flat organizational structure like Valve where you have no one to report to (well maybe Gabe Newell or the team leader for that project occasionally), but the culture is that everyone can manage themselves organically rather than having hierarchies and forced, useless reportings

Valve ships almost nothing, and the engineering quality is mediocre. Their game design is good, but subsidized by a unique early mover monopoly market owner position. There is almost nothing that can be generalized, and people who hold up their organization as an example to strive for are utterly clueless

> early mover monopoly

Well, while I do agree it is an early mover (I believe Steam is not the first to do online game platform, but it should indeed be the first one to use a CDN for the solution), I'm sorry that you have Ubisoft Connect, EA Origin, Epic Games Store, itch.io, GOG, Humble Bundle.

So you do have a choice. That renders the statement of it being a monopoly impossible. It's simply because every other platforms sucks.

In other words, it's like Nvidia. You choose Nvidia because AMD and Intel sucks, not because you don't have a choice.

> engineering quality is mediocre

You should see how CS2's smoke system works: https://www.youtube.com/watch?v=ryB8hT5TMSg

tl;dw: it is a voxel ray marching plus flood fill with Mie scattering. I'd like to point this out that is not only a really good game design, but it also requires ingenious engineering quality. I tried to improve upon it by adding octrees (or in other word, making it sparse voxel octree) but it doesn't seems to add much.

Oh and plus their new bomb shock wave system. I tried to reverse engineer it with vector field, but I couldn't come up with the ODE equations to recreate it. I believe it has to be some kind of field, because in a high energy explosion, it scatter through air into fluid-like structure and hence we can use some aerodynamics for simluation

Re: Why Software Factories Fail (or: harness engineering is not enough)

#274

Post by satvikpendem »

rom1504 wrote: post #132
satvikpendem wrote: post #127

Earlier quoted context omitted.

Why can they not do long term planning? They know what good software design looks like and can code accordingly. Using Fable for a greenfield project for example, it doesn't dump everything into one file but instead does indeed split out the work into logical units as needed.

Why is a very good question indeed! My guess is we have not figured out how to include long term enough tasks into the RL training. As to whether they can today, it's pretty easy to figure out that they can't by observing the difference between asking for very high level things like "please make me a website that will be very successful" vs sending it a detailed plans on the goals and many of the specifics and then h…

/goal and /workflow exists. As I said in another comment, models can do it, as in they have the capacity to do so. It's just the harness that needs to be wired up correctly.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#275

Post by satvikpendem »

dhorthy wrote: post #128
satvikpendem wrote: post #122

Earlier quoted context omitted.

GPT 5.6 Sol and Fable are similar step changes, not 5.5, but that's beside the point. If in the Markdown instruction files you ask them to periodically clean up the code or refactor where necessary then that's what they'll do. They don't explicitly do it because otherwise people, as they had for previous models, complain where models are asked to change one thing then they rewrite whole systems, so now the latest mod…

i guess to clarify my contention: generic prompting like "review this code" or "make the architecture better" will raise the floor but cannot come close to human-quality code without humans understanding what code exists and what to ask for. Obviously cursor and other labs have a stake in this going one direction, but I don't buy it, and I don't think you should either. In fact, "Rebuild sqlite from spec" has all the…

Fable writes code better than most humans can already so we've already surpassed human level coding. I think thinking otherwise is just coping for the job or industry.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#276

Post by kkapelon »

onion2k wrote: post #270
kkapelon wrote: post #233

Earlier quoted context omitted.

> Once you do all that the PR process is pointless manual PR reviews can catch things that llms currently miss. Examples are duplicated code, lack of unit tests or introducing security issues. None of these really break the build. So just requiring "do not break the build" is a very low barrier. Tests also are useless unless you have a smart system that runs the new test WITHOUT the changes and see it break. Most tea…

manual PR reviews can catch things that llms currently miss I didn't say LLMs though. I'm talking about the sort of code quality tooling that's been in use for decades - Sonarqube, Codacy, CodeClimate, etc for code quality, eslint for AST-based code rule checking (easily picks up duplication for example), Istanbul for code coverage, Wiz or Github Advanced Security for security. These are well known industry standards…

I am not familiar with all the tools you listed so please correct me if i am wrong, but all these will catch stuff that LLMs can potentially catch as well (if configured correctly).

They will not handle any architectural problems (i.e. this method is correct but doesn't belong on this package, or this method is called isX - but has side effects).

And I am pretty sure that none of them do what I am saying with the tests. i.e. run tests without the associated code change and see them fail. I am also pretty certain that none of them will understand problems with breaking backward compatibility (i.e., a fix that is correct that breaks the setup of all existing users).

In other words, it is possible today to create a PR that passes all checks, all security scans, all analysis tools, all test suite while still being wrong due to architecture, backwards compatibilty, wrong scope etc. So even before and even after LLMs a human is needed there.

PRs might become unhelpful as you say if in the future one of the two things we happen

1) LLMs have unlimited context so you can pass all source code plus all architectural designs them 2) We have a super smart analysis tool that is one level above what we have today (including llms)

We are not there yet.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#277

Post by clintonb »

canpan wrote: post #152
sathish316 wrote: post #149

I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…

It's even worse: Customer says I want X! One or more of the following can be true: - Customer really wants X - Customer doesnt actually need X at all - Customer wants Y not X but couldnt express himself - Customer B and C do NOT want X at all - Customer is just messing with you for no reason

https://xyproblem.info/

Re: Why Software Factories Fail (or: harness engineering is not enough)

#278

Post by zingar »

dhorthy wrote: post #79
zingar wrote: post #73

Enjoyed most of this but unconvinced by the program design part. If I see an agent writing function signatures or listing which functions to edit in a plan that tells me that I’ve given it too big a vertical slice. I always delete the code guesses. The thing that writes the code must always spend some time discovering where/what to write or it won’t have the right context. Or put another way: “decide first, act later…

one thing I probably didn't mention is we do the program design having already done an in-depth codebase research, with current patterns and architecture surfaced - that actually seeds every step of the flow including even the product part - but yes if you're working in very small slices then I think it's very feasible to skip program design and just review the code as you go, and resteer live. I do this all the time…

Do you think that the program design adds a lot on top of pointing out the current patterns?

I might be anchored by working with humans, but if I saw a plan for humans that included function signatures and what calls what I would say that is way too detailed. As a result I don't put it in my plans for AI either.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#279

Post by jambalaya8 »

bandrami wrote: post #192
jambalaya8 wrote: post #191

Earlier quoted context omitted.

What happens then, though? Is it good to have tens or hundreds of thousands of potentially angry marketing managers roaming around that were developed out of careers?

IDK, get Springsteen to record an aid concert for them?

I think you misconstrued my reply. I wasn't talking about that aspect.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#280

Post by forlorn_mammoth »

maerF0x0 wrote: post #84

I'm currently learning about Cost functions and Regression in the Machine learning course on coursera[1], and I cannot help but be struck by the similarities of how gradient descent seeks to minimize the error between the model and the training data, how agents do the same (far less mathematically) to seek an acceptable solution to software problems, and how light evolutionary pressures seem to guide species towards…

> how agents do the same (far less mathematically) to seek an acceptable solution to software problems,

except that there is no gradient towards 'better' software, at least not in a mathematical sense.

Post reply on HN