Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Software Factories Fail (or: harness engineering is not enough)

github.com

Post reply on HN
Print view View original

191–200 of 296 posts

Re: Why Software Factories Fail (or: harness engineering is not enough)

#191

Post by jambalaya8 »

bandrami wrote: post #189
tmikaeld wrote: post #186

Earlier quoted context omitted.

Marketing was a problem with apps even before AI, which, with today’s slopforests makes it even harder for consumers to see any of your trees even if they wanted to.

Right but if we're going to a small-batch bespoke development model then "marketing" more or less disappears because why would I pay someone to talk to an LLM when I can also talk to an LLM? The money goes from the end user to the lab in this model.

What happens then, though? Is it good to have tens or hundreds of thousands of potentially angry marketing managers roaming around that were developed out of careers?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#192

Post by bandrami »

jambalaya8 wrote: post #191
bandrami wrote: post #189

Earlier quoted context omitted.

Right but if we're going to a small-batch bespoke development model then "marketing" more or less disappears because why would I pay someone to talk to an LLM when I can also talk to an LLM? The money goes from the end user to the lab in this model.

What happens then, though? Is it good to have tens or hundreds of thousands of potentially angry marketing managers roaming around that were developed out of careers?

IDK, get Springsteen to record an aid concert for them?

Re: Why Software Factories Fail (or: harness engineering is not enough)

#193

Post by bandrami »

bandrami wrote: post #189
tmikaeld wrote: post #186

Earlier quoted context omitted.

Marketing was a problem with apps even before AI, which, with today’s slopforests makes it even harder for consumers to see any of your trees even if they wanted to.

Right but if we're going to a small-batch bespoke development model then "marketing" more or less disappears because why would I pay someone to talk to an LLM when I can also talk to an LLM? The money goes from the end user to the lab in this model.

I should expand that: the money goes from the end user to the lab and only in the cases where the marginal utility of the small app exceeds the cost.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#194

Post by preg_match »

dhorthy wrote: post #187
preg_match wrote: post #183

Earlier quoted context omitted.

The perfect poster boy is system scripts. They’re not even published usually, but being able to just generate < 500 line scripts for everything is a huge boon.

I agree this rocks I do this almost daily

It goes hand in hand with Linux desktops too, because everything is just so much more scriptable. That used to be intimating, user unfriendly, and maybe it still is. But with AI it enables so many new avenues.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#195

Post by [deleted] »

20k wrote: post #130
amannm wrote: post #42

Earlier quoted context omitted.

It seems weird to get angry about a change that is happening and will continue to happen due to what the market demands from software, which I believe will be the speed/ability to solve problems, rather than its own stewardship

I'm just here to do good software engineering and make good products. If people want to try and circumvent that process to deliberately make the product worse and slow development down, they're bad software engineers, its as simple as that Nobody thinks its a good idea to fully automate PRs, at least not anyone with software engineering experience. It slows down your ability to solve problems if your code base is an…

[deleted]

Re: Why Software Factories Fail (or: harness engineering is not enough)

#196

Post by antonvs »

d_silin wrote: post #38

My radical opinion is that LLMs are harmful for software development - they are the ultimate "goto" operator. All actual code should be written by a human developer. Instead, use them in adversarial mode - run QA scenarios using LLM agent as a substitute for end user to do bug discovery.

> All actual code should be written by a human developer.

This seems arbitrary. Why don’t you say the same thing about machine code? Developers use tools so they can avoid writing machine code. What is causing you to draw a line in the sand about use of tools?

The obvious answer is it’s just a function of the time period you grew up in, and a lack of willingness to adapt to change.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#197

Post by danpalmer »

latentsea wrote: post #185
danpalmer wrote: post #170

Earlier quoted context omitted.

But this is my point, they're not really controllable in this way. There are no fixed rules (like tests or linters) for small PRs, it's not good enough to do it by line count. A good PR tells a story to the reader that leaves them confident in the result. That means knowing the reader and how they think, and that is something LLMs are very bad at.

And you can't think of a way to bring an engineers mindset to solve that? If you leave it up to raw LLM outputs you're not going to get the results you're after. If you put on your engineering hat and think about your problem statement and what tools you can bring to bear on it you'll find you can likely craft a PR authoring skill and a set of evals for that skill that satisfies your requirements with a high enough d…

I'm not coming at this cold, I've worked on the skill for PR (CL) creation in the Google monorepo. I'm not talking about raw LLM outputs here. It's a hard problem that is fairly fundamental to the training material being used for LLMs today.

I'm sure we can improve things, I'm sure it will get better over time, but I'd say we're a small fraction of the way towards where we need to be, and trending in the wrong direction with long trajectory development creating bigger changes on average.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#198

Post by bandrami »

dboreham wrote: post #174
jpollock wrote: post #164

Earlier quoted context omitted.

There are also compliance reasons. SOX requires code review by a second party. If the reviews are automated, the development process isn't in compliance and that can get expensive in a hurry.

And of course these SOX-mandated second reviews were 99% of the time pure theater.

Sort of; it's like the itemizer systems at airport security. They're entirely theater (in the sense of "for show") but that show is itself important in that it makes entire categories of attacks significantly less feasible.

Sort of like how simply having a uniformed police officer on a street corner reduces street crime significantly even if the officer never interacts with anyone.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#199

Post by podocarp »

canpan wrote: post #152
sathish316 wrote: post #149

I call it the Intent-Implement-Quality problem. Software factories can implement anything given a one-liner requirement. That one-liner requirement can be a complete app/product, epic, feature, bug, design change or refactoring. But these one liner requirements are requirements coming from a human who has an intent or requirement or direction for the product to evolve in mind. Can Software factories manufacture inten…

It's even worse: Customer says I want X! One or more of the following can be true: - Customer really wants X - Customer doesnt actually need X at all - Customer wants Y not X but couldnt express himself - Customer B and C do NOT want X at all - Customer is just messing with you for no reason

Also quite common for me is like someone says X is good (usually a PM) but common sense says it's crap, and when you actually ask the user they also say it's crap. If only LLMs could email a bunch of people for second opinions.

Re: Why Software Factories Fail (or: harness engineering is not enough)

#200

Post by podocarp »

taneq wrote: post #179
andrewljohnson wrote: post #168

Earlier quoted context omitted.

You either need to quit or keep your head down, I can’t tell which.

If you need to actively keep your head down, you need to quit. Doesn’t matter whether ‘the problem’ is the employer, the employee, both, or neither. They’re a bad fit for each other.

I don't think so? It's very common in both large companies and small startups where pragmatism has a higher priority than so called elegance or whatever you call it the "artisan" feel. It's varying shades of "perfect is the enemy of good", where different cultures and business environments dictate how much a manager is going to tolerate your proposed refactoring. At least that's what I got from the parent lost.
Post reply on HN