Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The economics of software teams: Why most engineering orgs are flying blind

viktorcessan.com

Post reply on HN
Print view View original

151–160 of 299 posts

Re: The economics of software teams: Why most engineering orgs are flying blind

#151

Post by heroicmailman »

taurath wrote: post #41
bonesss wrote: post #25

Earlier quoted context omitted.

Ceding the premise that the AGI is gonna eat my job, my job involves reading the spec to be able verify the code and output so the there’s a human to fire and sue. There are five layers of fluffy management and corporate BS before we get to that part, and the AGI is more competent at those fungible skills. With the annoying process people out of the picture, even reviewing vibeslop full time sounds kinda nice… Feet u…

There’s gonna be one guy in charge of you, and he’s going to expect you to be putting out 20x output while thanking him for the privilege of being employed, assuming all goes the way every management team seems to want I dont think this will happen because AI has become a straight up cult and things that are going well don’t need so many people performatively telling each other how well things are going.

> There’s gonna be one guy in charge of you, and he’s going to expect you to be putting out 20x output while thanking him for the privilege of being employed, assuming all goes the way every management team seems to want

A perfect summation.

Re: The economics of software teams: Why most engineering orgs are flying blind

#152

Post by skrebbel »

pron wrote: post #95

> A messy codebase is still cheaper to send ten agents through than to staff a team around People who say that haven't used today's agents enough or haven't looked closely at what they produce. The code they write isn't messy at all. It's more like asking the agent to build a building from floorplans and spec, and it produces everything in the right measurements and right colours and passes all tests. Except then you…

> the art is load-bearing

This is beautiful

Re: The economics of software teams: Why most engineering orgs are flying blind

#153

Post by chermi »

pron wrote: post #95

> A messy codebase is still cheaper to send ten agents through than to staff a team around People who say that haven't used today's agents enough or haven't looked closely at what they produce. The code they write isn't messy at all. It's more like asking the agent to build a building from floorplans and spec, and it produces everything in the right measurements and right colours and passes all tests. Except then you…

This just sounds like incomplete specs to me. And poor testing.

Re: The economics of software teams: Why most engineering orgs are flying blind

#154

Post by gwerbin »

chermi wrote: post #153
pron wrote: post #95

> A messy codebase is still cheaper to send ten agents through than to staff a team around People who say that haven't used today's agents enough or haven't looked closely at what they produce. The code they write isn't messy at all. It's more like asking the agent to build a building from floorplans and spec, and it produces everything in the right measurements and right colours and passes all tests. Except then you…

This just sounds like incomplete specs to me. And poor testing.

"Incomplete specs" is the way of the world. Even highly engineered projects like buildings have "incomplete specs" because the world is unpredictable and you simply cannot anticipate everything that might come up.

Re: The economics of software teams: Why most engineering orgs are flying blind

#155

Post by pron »

chermi wrote: post #153
pron wrote: post #95

> A messy codebase is still cheaper to send ten agents through than to staff a team around People who say that haven't used today's agents enough or haven't looked closely at what they produce. The code they write isn't messy at all. It's more like asking the agent to build a building from floorplans and spec, and it produces everything in the right measurements and right colours and passes all tests. Except then you…

This just sounds like incomplete specs to me. And poor testing.

It isn't. Anthropic tried building a fairly simple piece of software (a C compiler) with a full spec, thousands of human-written tests, and a reference implementation - all of which were made available to the agent and the model trained on. It's hard to imagine a better tested, better-specified project, and we're talking about 20KLOC. Their agents worked for two weeks and produced a 100KLOC codebase that was unsalvageable - any fix to one thing broke another [1]. Again, their attempt was to write software that's smaller, better tested, and better specified than virtually any piece of real software and the agents still failed.

Today's agents are simply not capable enough to write evolvable software without close supervision to save them from the catastrophic mistakes they make on their own with alarming frequency.

Specifically, if you look at agent-generated code, it is typically highly defensive, even against bugs in its own code. It establishes an invariant and then writes a contingency in case the invariant doesn't hold. I once asked it to maintain some data structure so that it could avoid a costly loop. It did, but in the same round it added a contingency (that uses the expensive loop) in the code that consumes the data structure in case it maintained it incorrectly.

This makes it very hard for both humans and the agent to find later bugs and know what the invariants are. How do you test for that? You may think you can spec against that, but you can't, because these are code-level invariants, not behavioural invariants. The best you can do is ask the agent to document every code-level invariant it establishes and rely on it. That can work for a while, but after some time there's just too much, and the agent starts ignoring the instructions.

I think that people who believe that agents produce fine-but-messy code without close supervision either don't carefully review the code or abandon the project before it collapses. There's no way people who use agents a lot and supervise them closely believe they can just work on their own.

[1]: https://www.anthropic.com/engineering/building-c-compiler

Re: The economics of software teams: Why most engineering orgs are flying blind

#156

Post by pphysch »

chermi wrote: post #153
pron wrote: post #95

> A messy codebase is still cheaper to send ten agents through than to staff a team around People who say that haven't used today's agents enough or haven't looked closely at what they produce. The code they write isn't messy at all. It's more like asking the agent to build a building from floorplans and spec, and it produces everything in the right measurements and right colours and passes all tests. Except then you…

This just sounds like incomplete specs to me. And poor testing.

A sufficiently complete spec is indistinguishable from source code.

Re: The economics of software teams: Why most engineering orgs are flying blind

#157

Post by pphysch »

rpdillon wrote: post #117
RALaBarge wrote: post #104

Earlier quoted context omitted.

The more I work with AIs (I build AI harnessing tools), the more I see similarities between the common attention failures that humans make. I forgot this one thing and it fucks everything up, or you just told me but I have too much in my mind as context that I forget that piece, or even in the case of Claude last night attesting to me while I am ordering it around that it cannot SSH into another server but I find it…

Right, but when humans are writing the code, they have learned to focus on putting downward pressure on the complexity of the system to help mitigate this effect. I don't get the sense that agents have gotten there yet.

Big business LLMs even have the opposite incentive, to churn as many tokens as possible.

Re: The economics of software teams: Why most engineering orgs are flying blind

#158

Post by tedggh »

taurath wrote: post #41
bonesss wrote: post #25

Earlier quoted context omitted.

Ceding the premise that the AGI is gonna eat my job, my job involves reading the spec to be able verify the code and output so the there’s a human to fire and sue. There are five layers of fluffy management and corporate BS before we get to that part, and the AGI is more competent at those fungible skills. With the annoying process people out of the picture, even reviewing vibeslop full time sounds kinda nice… Feet u…

There’s gonna be one guy in charge of you, and he’s going to expect you to be putting out 20x output while thanking him for the privilege of being employed, assuming all goes the way every management team seems to want I dont think this will happen because AI has become a straight up cult and things that are going well don’t need so many people performatively telling each other how well things are going.

If a SWE could truly output 20x their effort, that person would probably be better at freelancing or teaming up with another SWE. If something can be automated away to AI is Project Management. Also, there has to be a point where delivering more and faster code doesn’t matter, because the choke points are somewhere else in the Project Life Cycle, say waiting for legal, other vendors, budgets, suppliers, etc, so productivity could max out at say 3X, after which, unless you have a strong pipeline of work, your engineers will be sitting around waiting for the next phase of the project to start.

Re: The economics of software teams: Why most engineering orgs are flying blind

#159

Post by pphysch »

tripledry wrote: post #111
iamflimflam1 wrote: post #21

Earlier quoted context omitted.

Very much like humans when they drown in technical debt. I think the idea that a messy codebase can be magically fixed is laughable. What I might believe though is that agents might make rewrites a lot more easy. “Now we know what we were trying to build - let’s do it properly this time!”

I'm wondering how much value there is in a rewrite once you factor in that no one understands the new implementation as well as the old one. Not only is it difficult to verify, but also the knowledge your team had of your messy codebase is now mostly gone. I would argue there is value in knowing your codebase and that you can't have the same level of understanding with AI generated code vs yours.

The point of a rewrite is to safely delete most of that arcane knowledge required to operate the old system, by reducing the operational complexity of it.

Re: The economics of software teams: Why most engineering orgs are flying blind

#160

Post by bitwize »

ThunderSizzle wrote: post #86
ben_w wrote: post #56

Earlier quoted context omitted.

> my job involves reading the spec to be able verify the code and output so the there’s a human to fire and sue. So, you're the programmer (verify code) and the QA (verify output) and the project manager (read the spec)?

That's the difference between programming and software engineering. A software engineer should be able to talk directly to customers to capture requirements, turn that into spec sheet, create an estimate and a bunch of work items, write the whole system (or involve other developers/engineers/programmers to woek on their work items), and finally be able to verify and test the whole system. That entire role is software…

Careful with that though. The guy whose entire job is to "take requirements from the customers and bring them to the engineers" really does get awful tetchy if the engineers start presuming to fill his role. Ask me how I know.
Post reply on HN