Software Estimation Is a Losing Game (2014)

When you decompose a larger task into smaller ones, that's all still true for each of the smaller ones, right?

But you must remember to add the errors! And that is precisely why overall confidence may not likely improve with decomposition vs. an aggregate, or coarse, high level estimate of the whole thing.

And this is true for more than software. Why?

New tasks can be broken down into two coarse buckets: done before and novel.

Novel things really are novel! It's important to recognize this case and qualify it. Progress might look like:

week 1 - 10 percent done, week 2 - 10 percent done, week 3 - 15 percent done, week 4 - 50 percent done! Yes!, week 5 - 5 percent done... everybody take a drink, week 6 - 40 percent done,

etc...

Because it's novel, there are new dynamics in play, and even the best planning may not account for, and actually usually does not account for, things that just come up and that must be dealt with.

For the done before case, it's better as a lot is known, but there are still many factors out of the team control. Updates, changes, regressions, unfound bugs, all contribute to variance. The more the organization completely owns, the less this may be true, but it's always somewhat true.

The pragmatic person might classify everything as having some percentage of novel, and break it down into known novel and unknown, "it just hit us!" type novel and go from there and likely be better off when having to manage expectations.

Another way to think of this, depending on it being software or something else, is production vs "R&D" and the research and development part just isn't cog / production like. Time and materials estimates, if they are even given, must be considered accordingly.

If they are not, expectations do not match realities, and now there is a management and potentially a funding problem in addition to the primary problem needing a solution.

It gets even more complicated because, at least in my experience, software processes are really subtle and it is extremely easy to appear to be "doing the right thing", while at the same time failing miserably at it. Throw in the power politics, the bad actors who don't really care if something is successful or not (as long as they can extract money from it), and naive developers who are absolutely sure they know what to do but really don't, and it makes it very difficult.

It is difficult to be the one instigating process change. Such a role is fraught with danger. You will be attacked by the power politics people. You will be attacked by the money extracting people. You will be attacked by the naive developers who think they know better than you. You almost certainly will not be thanked by anyone (except very far down the road if you somehow manage to succeed... and even then probably nobody will remember it was you who started the ball rolling). And finally, because it is incredibly hard and the necessary skills are very difficult to acquire, you will almost certainly fail many, many times before you ever succeed.

Even worse, it's not something you can write down and say "Do X,Y, and Z and everyone will be a happy family working hard to ensure success". It is something that requires dedication to understanding every single individual who is involved in the development. You have to sit and listen to them, understand their goals and help them meet those goals (even if they aren't "Make a successful software product" and even if you personally think the person is unbearably awful). You have to devote yourself to finding common ground and bringing everyone together.

Don't get me wrong. The processes and practices are crucial to a successful organizational change. These practices, if well chosen, will support you as you influence, cajole, flatter, criticize and motivate the people around you. It is next to an impossible job and without these tools, you will almost certainly fail.

It is unfortunate that people dramatically underestimate the difficulty of cultural change in an organization. They write about the easy bits (the processes that support the change) and sweep the hard bits under the carpet. To be fair, I'm not sure what you would say that wouldn't sound like "business BS" to most engineers, though.

Anyway, most organizations do not have many people conscientious (possibly read "stupid") enough to attempt real, lasting cultural change. The vast majority that do try are often incredibly naive and lack all of the basic skills necessary to be successful. The end result is unfortunately all too predictable.

To anyone who is really serious about building an amazing team, I encourage you to suffer the slings and arrows of outrageous management and to work on the people skills that will eventually bring you what you want. It sucks for a very long time, but as you get a better understanding, it can get better.

Also, how did you come to the conclusion that "project management doesn't work" is a historical fact?

Also also, if your teams' estimates are a farce you might consider starting to track time spent on projects and build up a base of real world numbers you can use as guidelines for future estimates. It worked wonders for us.

I am not saying project mgmt does not work, I am saying the plan the task, do the task approach is not appropriate for software development, at a given level of granularity.

If you ever watch Grand Designs, they start with a design, a plan, a schedule, and it all goes out the window ten minutes later. This is not that being on top of the process will not give you control, but that relying on estimates and plans as a means of control is worthless - and that's for something people have been doing for hundreds of years

Now, I don't work in software engineering, I work in traditional engineering, but I find the lack of any formal Risk Assessment, Management or Lessons Learned in any Project Management methodology/framework a little scary.