Why Development Teams Struggle to Deliver on Time, on Budget, or at All • Hacker News Board

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#61

Post by sonnyblarney » Tue, Jun 05, 2018, 2:58 PM UTC

kareemm wrote: post #7

In my experience talking to hundreds of software engineering managers, running a software project management business, and being involved on software dev teams for 20+ years, there are two reasons why teams don’t ship or ship poor software. 1. Poor engineering planning. The way to do this well is to break down a feature until they’re at about half-day sizes tasks. Identify any obvious risks or ambiguities with the te…

I have to respectfully disagree with #1.

Software is hard because we 'don't know the unknowns'.

It's not like building a building - if it were, we could just schedule it down to each detail. We would be still all be using 'waterfall'.

If we knew how to write the module, we'd just do it, test it, and voila.

But most of our time is spent 'figuring stuff out'. Oh, product has asked for a tiny change (support a specific kind of font), but our libraries don't support that, meaning major possible architectural changes.

The other major unspoken factor is that requirements change. In fact, this is the #1 thing, really. Requirements change because customer's needs change, because business changes because our understanding of the market changes, etc., and this is a reality of our world. So the nature of how we're going to adapt is difficult to predict.

This is why the #1 predictor of outcomes is the quality of developers on the team i.e. experience, intelligence, know-how of the domain.

If you're doing something where there are very few unknowns, well then it should be possible to schedule within reason.

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#62

Post by dredmorbius » Tue, Jun 05, 2018, 2:58 PM UTC

gonzo41 wrote: post #16

jhall1468 wrote: post #3

In my experience, deadlines set by higher-ups long removed from (or never having) the technical chops to be determining the deadlines in the first place. VP's and Directors are often the ones dictating the direction, which is great, but then also introducing deadlines, with helpful input from Directors who ALSO haven't touched code in many years. Generally that results in one of two things: * Product delivered on-tim…

Software is up there as the most complex thing ever created. People have been putting up houses for thousands of years, And modern big bridges for hundreds. Big software is ~40-60 years old. Its hard to have rules of thumb and best practice that will deliver the 'on time outcome'. Give it another 100 years. Most dev work right now is still just attempting to deliver correct software. let alone on time or on budget.

The Linux kernel is several times more complex than the Space shuttle. Order of magnitude as I recall.

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#63

Post by tootie » Tue, Jun 05, 2018, 3:00 PM UTC

This is literally what agile is for. Step 1 (and this is the hardest part) is to negotiate the need for either a flexible scope or deadline. Accept that you're estimate for a fixed scope will just never be accurate and just make room to adjust. Putting "buffer" into your estimate is also not the right approach because it's planning for failure. Flexible scope means you have work to fill up what would go into your buffer that you can launch without if you need to, but would be very nice to have.

Create your high-level backlog and do MoSCoW prioritization. Figure out your "musts", "shoulds", "coulds", "wonts". Now apply some estimates to your features and add 10% for unforeseen growth. Estimate velocity based on team size and now you've got a date when you could conceivably hit your musts, shoulds, coulds. Set your "deadline" if you must somewhere deep in the coulds. If your musts go over, you are still able to launch an MVP. If things go well, you can start delivering non-musts.

Adjust your plan every sprint based on actual velocity.

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#64

Post by DanHulton » Tue, Jun 05, 2018, 3:00 PM UTC

ttul wrote: post #11

jhall1468 wrote: post #3

In my experience, deadlines set by higher-ups long removed from (or never having) the technical chops to be determining the deadlines in the first place. VP's and Directors are often the ones dictating the direction, which is great, but then also introducing deadlines, with helpful input from Directors who ALSO haven't touched code in many years. Generally that results in one of two things: * Product delivered on-tim…

Having worked in the trenches as a software engineer and also as a CEO has given me respect for the VP level as well as the engineers. When the VP sets a deadline, it’s driven by the survival of the company - aka release by June or we can’t make our numbers and have to do layoffs. When the engineers push back, they often do so without an appreciation of the business reality, aka those numbers pay their salaries. Engi…

Ideally, yes, a deadline is set for a specific business reason. But I cannot count the number of times a deadline has been set for a project that is just utterly arbitrary, proven by the project releasing successfully on the deadline and then being immediately abandoned and forgotten about. Feature launches without marketing, website redesigns without fanfare, architecture refactoring that sits on the shelf for a month before actually being deployed.

Not that I'm claiming that all executive decisions are clueless, most people are just trying to do a good job with limited information, and mistakes happen. But you run into enough executives without respect that runs in the other direction, and you start to question all deadlines.

(Which isn't necessarily a bad thing. In a healthy environment, questioning the feasibility of a deadline can lead to appropriate descoping of overreaching features, reduction of risk, etc. But the situations that lead to this kind of suspicion are rarely healthy to begin with.)

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#65

Post by JohnL4 » Tue, Jun 05, 2018, 3:01 PM UTC

golergka wrote: post #14

> This is where System 2 comes in—if we performed a more thorough analysis, these factors would have been considered in our answer. Then it would be clear that it’s much more likely to take 20 or 30 minutes to run to the store instead of 10. I have encountered an article (can't find the link now, sadly) that claimed that when developers gave estimations, breaking down tasks to sub-tasks actually had a reverse correla…

Maybe because people over-rely on a fine-grained breakdown? When I try that, I miss big stuff. Kind fits in with that Fibonacci estimation from Agile, eh?

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#66

Post by jcadam » Tue, Jun 05, 2018, 3:03 PM UTC

golergka wrote: post #14

> This is where System 2 comes in—if we performed a more thorough analysis, these factors would have been considered in our answer. Then it would be clear that it’s much more likely to take 20 or 30 minutes to run to the store instead of 10. I have encountered an article (can't find the link now, sadly) that claimed that when developers gave estimations, breaking down tasks to sub-tasks actually had a reverse correla…

> I have encountered an article (can't find the link now, sadly) that claimed that when developers gave estimations, breaking down tasks to sub-tasks actually had a reverse correlation to their accuracy. In other words, their first gut reactions were actually better than estimations given after going in-depth through all the details and sub-tasks.

I've noticed this as well. The last few years my 'gut' estimates have tended to be more accurate (and larger) - but since I can't support them with anything other than "Trust me, I've been writing software for over a decade", they're never taken seriously.

My guess would be that breaking down a complex software system into sub-tasks (esp. when you are breaking things down within modules) that are too fine grained fails to capture the inter-dependencies between sections of the code-base. So, you complete task A and move onto to task B, then realize you have to revisit the code you wrote during task A because task B is dependent on it and some of your early assumptions were imperfect. And then you need to fix the unit tests you wrote for task A.

And so on, and so forth.

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#67

Post by gonzo41 » Tue, Jun 05, 2018, 3:03 PM UTC

UK-Al05 wrote: post #46

Having been through multiple month long planning "phases", that were eventually shown to be wildly inaccurate I disagree. This is exactly what we used to do in the waterfall days. It didn't work. The only way that's been compatible with me is to build something really small but valuable. So small that its hard to be disastrously wrong. Once you've released that value, build upon it. Stakeholders tend be much happier…

This for the win! Always be wary when they ask for single sign on in the first sprint!

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#68

Post by ThomPete » Tue, Jun 05, 2018, 3:04 PM UTC

There are ways to deliver on time & budget but most developers are not going to like it neither are their "clients".

Use the frameworks exactly as they are intended, don't try to invent new solutions that aren't native to the framework.

Anything that takes you outside the beaten path in development is going to be a potentially infinite black box.

In other words, a lot of software engineering can't be put in timeboxes because it's actually R&D more than it's development and where each little step forward can add a potentially infinite amount of new tasks to be done or problems to be solved. Add to that the constant need to update, upgrade, improve, re-design and you know it's just not doable.

So the primary problem IMO is that we think about a lot of development as if it's something that can be put in boxes. Some can of course and the better and more solid the team becomes to better they are but the teams who struggle are mostly struggling because the expectations for what they are actually doing (inventing problem-solving) isn't matching up with what they are being paid to do (build)

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#69

Post by commandlinefan » Tue, Jun 05, 2018, 3:05 PM UTC

Aaargh20318 wrote: post #43

The reason that planning software development is bullshit is that you simply cannot know all the little details because in software development you're always doing something you've never done before (because if you did you could just copy-paste your previous work). To use the 'going to the supermarket for milk' example. I could make a fairly accurate estimate for that because I've gone to the supermarket hundreds of…

And TFA's suggestion (use our magical time tracking software) isn't going to solve that problem, either.

Re: Why Development Teams Struggle to Deliver on Time, on Budget, or at All

#70

Post by shadowmint » Tue, Jun 05, 2018, 3:05 PM UTC

kqvamxurcagg wrote: post #17

kareemm wrote: post #7

In my experience talking to hundreds of software engineering managers, running a software project management business, and being involved on software dev teams for 20+ years, there are two reasons why teams don’t ship or ship poor software. 1. Poor engineering planning. The way to do this well is to break down a feature until they’re at about half-day sizes tasks. Identify any obvious risks or ambiguities with the te…

How do you create this detailed plan? Are there any tools or frameworks?

There are literally hundreds of methods and tools; it doesn’t matter which one(s) you pick.

I’m quite serious.

The thing you need to realize is that you’ll get it wrong, no matter the tool... until you invest time in reviewing past estimates and plans.

People think because they’re engineers they can just magically make estimates... but its a skill. If you don’t have a feedback loop to measure your progress and adjust your actions, you’ll literally never improve at it.

...that’s why even ‘Senior’ developers and ‘technical leads’ are often terrible at this; they’ve simply never bothered to seriously make an effort to learn how to do it.