Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The economics of software teams: Why most engineering orgs are flying blind

viktorcessan.com

Post reply on HN
Print view View original

291–299 of 299 posts

Re: The economics of software teams: Why most engineering orgs are flying blind

#291

Post by ericyd »

> their value is, at least in principle, calculable

I feel this article does not spend enough time investigating the challenges of directly measuring the financial output of an engineering team. I agree it is theoretically possible, but I don't think the full answer is that people got lazy on cheap capital and didn't care enough to measure. I think it would be exceedingly difficult to put a dollar amount on the monthly output of most engineering teams due to the variety of tasks they cover, and the extreme challenge of knowing exactly why your customers are behaving a certain way. If you get 1000 more signups in a month, is that directly attributable to the engineering team's output? If anyone could have been concretely answering that question this whole time, I don't think they would have been ignoring the metrics.

Re: The economics of software teams: Why most engineering orgs are flying blind

#292

Post by Anamon »

leokennis wrote: post #3

> The obvious objection is that code produced at that speed becomes unmanageable, a liability in itself. That is a reasonable concern, but it largely applies when agents produce code that humans then maintain. Agentic platforms are being iterated upon quickly, and for established patterns and non-business-critical code, which is the majority of what most engineering organizations actually maintain, detailed human fam…

Yeah, that paragraph really betrayed the author's ignorance of software development. At the very least, it proves that they have no hands-on experience with LLM-assisted development.

Getting these tools to "understand", or be able to generate good results in a codebase, is not a function of the number of agents or the time you let them run. Much rather, if the tools fail to produce anything useful after a few minutes, you can bet your ass that they're not going to work better after hours, or days. If they come up with a mess, and your reaction is to just let them work on it for a few days, I can confidently predict what you'll end up with.

They come close to grasping what we've learned about where these new tools are useful and where they aren't, only to end up falling for the pretty words these generators use to lipstick their turds. As right as they may be about the financial considerations, there are going to be some very uncomfortable bills to pay for those who share this belief in the magical abilities of LLMs.

Re: The economics of software teams: Why most engineering orgs are flying blind

#293

Post by Anamon »

rstuart4133 wrote: post #269
llmssuck wrote: post #238

Earlier quoted context omitted.

Fair enough, but I think that never really worked all that well. What I mean is that the term "programming" would then essentially cover anything and everything that can be put into an algorithm of some sort. Neutrino detection, user management dashboards, CRUD APIs, basically everything is programming. It would explain a lot of misunderstanding between "programmers" though.

It's true that we often split software engineering into two pieces: doing the design and implementing the code. In fact I often prefer to think of it as computer systems engineering because the design phase often goes well beyond software. You have to think about what networks are used, what the hardware form factor should be used, even how many check digits should be put on a barcode. But then you go onto say this:…

They can help with those tasks because there are decades of published research on them. I don't think LLMs change anything here. Even before LLMs, it wouldn't have been efficient to ignore readily available lessons from those who solved similar problems before. As you put it, LLMs can be great search engines. It's not that they changed the task.

Whether you have to solve a problem that is hard today because there aren't many available resources, or something well discussed that you can research with Google or an LLM, I don't think it changes anything about their argument that once you know what to do, actually turning it into working code is comparatively mundane and unchallenging, and always has been to some degree.

Re: The economics of software teams: Why most engineering orgs are flying blind

#294

Post by Anamon »

wmwragg wrote: post #231
Aurornis wrote: post #201

Earlier quoted context omitted.

> which is that the actual programming is not the hard part. Figuring out what exactly needs programmed is the hard part. I’m growing tired of this aphorism because I’ve been in enough situations where it was not true. Some times the programming part really is very hard even when it’s easy to know what needs to be built. I’ve worked on some projects where the business proposition was conceptually simple but the whole…

Yep, I think people who repeat this aphorism essentially equate programming with typing, or as you say just connecting existing bits together. Programming is the working out how to get a computer to perform some task, not just the typing, it's the algorithms, the performance balancing, the structuring, the integration etc.

I think that's fine and probably kind of their point, no? Because that other part before the typing is also the thing that LLMs are woefully shit at, unless it's a common and solved problem with 1000 published examples.

Re: The economics of software teams: Why most engineering orgs are flying blind

#295

Post by rstuart4133 »

Anamon wrote: post #293
rstuart4133 wrote: post #269

Earlier quoted context omitted.

It's true that we often split software engineering into two pieces: doing the design and implementing the code. In fact I often prefer to think of it as computer systems engineering because the design phase often goes well beyond software. You have to think about what networks are used, what the hardware form factor should be used, even how many check digits should be put on a barcode. But then you go onto say this:…

They can help with those tasks because there are decades of published research on them. I don't think LLMs change anything here. Even before LLMs, it wouldn't have been efficient to ignore readily available lessons from those who solved similar problems before. As you put it, LLMs can be great search engines. It's not that they changed the task. Whether you have to solve a problem that is hard today because there are…

> They can help with those tasks because there are decades of published research on them.

No.

I suspect the underlying meme behind this is "LLMs can't think, they are just stochastic parrots". If you define an LLM to be the thing the model builders train by ingesting and compressing information on the internet, in a way that yields a very convenient if somewhat unreliable search engine, then that's true. But if you define them that way, the modern models like Opus, Gemini, GLM and so on aren't pure LLMs, and haven't been for a while. They still have an LLM inside them, but have other things bolted on like attention heads, context windows, and Chain of Thought. These things outperform humans at some tasks that require "thinking".

Glasswing is an example. Before Glasswing human "researchers" could find a solution to a particular problem, the "lets find an exploit in this software" problem, as a particular rate. Now with Glasswing, it's orders of magnitude faster than that rate. The same thing has happened in other areas, like protein folding and weather prediction. These models aren't reproducing something they've seen before. They are producing things we consider new - like creating an exploit for a vulnerability they discovered.

With all the evidence around if you are still hanging onto the idea these models can't help with many tasks by doing what we call "thinking", you are in denial.

Re: The economics of software teams: Why most engineering orgs are flying blind

#296

Post by latentsea »

pron wrote: post #198
latentsea wrote: post #191

Earlier quoted context omitted.

They can work really well if you put sufficient upfront engineering into your architecture and it's guardrails, such that agents (nor humans) basically can't produce incorrect code in the codebase. If you just let them rip without that, then they require very heavy baby-sitting. With that, they're a serious force-multiplier.

They don't work really well even on relatively small things and even with a virtually impractical upfront engineering: https://news.ycombinator.com/item?id=47752626 They just make a lot of mistakes that compound and they don't identify. They currently need to be very closely supervised if you want the codebase to continue to evolve for any significant amount of time. They do work well when you detect their mistakes a…

We've done the impractical upfront engineering, and they're working well for us :)

Re: The economics of software teams: Why most engineering orgs are flying blind

#297

Post by theK »

bitwize wrote: post #160
ThunderSizzle wrote: post #86

Earlier quoted context omitted.

That's the difference between programming and software engineering. A software engineer should be able to talk directly to customers to capture requirements, turn that into spec sheet, create an estimate and a bunch of work items, write the whole system (or involve other developers/engineers/programmers to woek on their work items), and finally be able to verify and test the whole system. That entire role is software…

Careful with that though. The guy whose entire job is to "take requirements from the customers and bring them to the engineers" really does get awful tetchy if the engineers start presuming to fill his role. Ask me how I know.

Are you talking about the "engineer talked to a customer and now both are mad at each other" trope?

While I have seen this happening it usually has nothing to do with engineers and more with that fact that talking to customers and identifying requirements is a task that requires respect and practice to become good at. Procentually I've seen more junior MBAs alienate customers than I have engineers seen do it.

Re: The economics of software teams: Why most engineering orgs are flying blind

#298

Post by ytoawwhra92 »

kevin_nisbet wrote: post #147

I'm not commenting too much on the details of the article, but the premise does resonate with me. I would argue all the engineering teams I've been on do not spend enough time thinking about how much a piece of work will cost to execute, and whether it will generate a return. I suspect this is most apparent on things like meeting culture. Something happens and all of a sudden there is another recurring meeting on the…

> I would argue all the engineering teams I've been on do not spend enough time thinking about how much a piece of work will cost to execute, and whether it will generate a return.

These things are almost never the responsibility of engineering teams, nor should they be. Engineers read about these things and get it into their head that they should be worrying about them. In effective organisations, these decisions are made elsewhere, by people better qualified to make these determinations. It's an ineffective organisation that leaves these sorts of decisions to their engineering teams.

Re: The economics of software teams: Why most engineering orgs are flying blind

#299

Post by pron »

latentsea wrote: post #296
pron wrote: post #198

Earlier quoted context omitted.

They don't work really well even on relatively small things and even with a virtually impractical upfront engineering: https://news.ycombinator.com/item?id=47752626 They just make a lot of mistakes that compound and they don't identify. They currently need to be very closely supervised if you want the codebase to continue to evolve for any significant amount of time. They do work well when you detect their mistakes a…

We've done the impractical upfront engineering, and they're working well for us :)

Since Anthropic weren't able to make them work even for something as simple and familiar as a C compiler then I would guess that:

1. You're supervising the agents closely, or

2. Your projects are very simple - simpler even than a C compiler, or,

3. They're not really working well; the catastrophic problems just haven't surfaced yet.

Post reply on HN