Live data from Hacker News

Why are software development estimates regularly off by a factor of 2-3?

michaelrwolfe.com

101–110 of 173 posts

Re: Why are software development estimates regularly off by a factor of 2-3?

#101
post #34

I love this post; it conveys the feel of the experience so well. One of my big aha moments about estimation was a bit in McConnell's Rapid Development . He pointed out that most estimates get made with executives pressuring for short numbers. When you iterate a few times with that, you end up with the smallest number that developers can't absolutely prove is impossible. If you draw out a bell curve of probable comple…

I once read an argument that went something like this: if you're actually going to go to the trouble to build something, its expected return to the business should be so large that how long it takes (within reason) shouldn't matter. Conversely, if a project is only economic to undertake if it can be done within a certain amount of time, it shouldn't be done at all.

I find that I often don't have a good feel for how long something is going to take until I'm about 1/4 the way through the project. If I absolutely had to produce accurate, high-confidence estimates, I would just do 1/4 of the work first, then estimate the rest.

As to why estimating is so hard, here's how I like to think about it. Tasks form a tree: each project has a number of steps, each of which itself breaks down into sub-steps, and so on. We tend to estimate by thinking about the tasks near the root of the tree; but the actual time required is proportional to the number of leaves -- tasks we can implement without further breakdown. Without planning the entire project out in detail it's hard to know what that number is. But to plan to that level of detail requires making many design decisions; and researching and making those decisions is much of the work of the project.

The bottom line is, good estimates are expensive to produce, and my experience aligns with yours: it's rarely worth the trouble.

Re: Why are software development estimates regularly off by a factor of 2-3?

#102
post #44
post #18

So the article tries to describe the hidden complexity of software with a hiking analogy but then it leaves a huge part missing. Let's extend the analogy somewhat... this isn't your first hiking trip. In fact, you've been on dozens and dozens of hiking trips. You're an experienced hiker. Yet, why do you continue to give meaningless as-the-crow-flies estimates of how long it will take you to get to your destination? T…

I'm glad you got better at it, but I don't think software estimates can ever be very good. Nor should they be. That a project is estimatable has to mean that novelty is low and predictability is high. But low novelty is a sign of duplication, which should be factored out into things like libraries, frameworks, and self-service tools. High predictability is either a sign of duplication or it's a sign that people have…

I do flat-rate projects like this using a library that I have to fill in gaps myself. Even in the context of very high unpredictability, I have to make good predictions else suffer low compensation exactly like the parent poster did.

I like crow-flies vs hiking analogy. I'd like to add upon it for so-called "green field" programming as it relates to getting cost predictions correct and understanding programmer time growth in relation to program size. Some parts are linear. Some parts are exponential.

Before I go on talking about "hello-world" space, note that I do architecture and really like architecture, big-picture solutions etc but I try only to solve what is apparent and then iterate on it whenever I notice two pieces tangling. There are excellent wins to be made here, but a working program itself constitutes part of the information necessary to arrive at the final architectural decisions. This is why even on FOSS code, I try not to sweat the architecture on the first pass -- when I do, dead code results. It calls out to me that I have solved something not relevant to the emergent implementation.

Once I've decided what the API might look like, what data structures have to be accessible to what, and what the minimum program states are, I try to stay in what I call "hello world" space. The idea is that the code I'm writing is never doing anything more than one problem at a time, line by line if I don't know exactly what a line does.

This is incredibly efficient because I deliberately break my problem into something testable and well understandable at every step. However, it's impossible to write production code this way. Not only is test feedback, (printlining and frequently more complex testing) not part of the final behavior, but more importantly, the elimination of concurrence of many states, the overlapping of problem workflows in execution sequence, implicitly says that "hello world" space problems are not production code. They have eliminated some functionality or consideration so as to make "hello world" solutions utterly oblivious to concurrent states or processes.

However, two "hello world" problems that need some coordination themselves creates a third "hello world" problem to implement their coordination. The third problem isn't apparent until the first two are explicitly solved. This implies hiking from the article.

Solving "hello world" problems brings more of the problem domain into "hello world" space. The concurrence problem itself will become apparent as a "hello world" problem when it has manifested itself. If there are two states that have been independently implemented but can overlap in execution, there is at least one set of logical statements for dealing with the maximum concurrence of the two states. Growth for N states is exactly an NxN truth table unless some of the states are sparse. In the worst-case, for N "hello world" solutions, one layer of abstraction where all states depend on all other states will result in N^2 logical blocks. While in practice this concurrence is usually much lower, the growth in abstraction is exponential if one is to keep breaking each problem down into an independently writable, testable piece.

So my conclusion is that growth is pretty much linear in proportion to writing "hello world" problems and exponentially proportional how many layers of integration are necessary. Slamming code down ichi-geki style is linear for problems that don't have a potential for overlap and geometric for ones that do. Code that eliminates duplication of routines through logic creates them in implementation, but the payoff is a net gain in many ways, so of course it still makes sense to do it.

As for making predictions, while running my internal monte-carlo, the concurrence of states has a higher influence on the outcome sometimes than the raw code size I expect to come out in the end. Linear and exponential growth occur in different areas. Recognizing that each integration incurs smaller linear component, and that each abstraction layer involves N integrations of solutions, I think it's pretty achievable to have a damned good estimate. The exception is if scope -must- creep and this is only determined in the middle of a project. Scope creep gets paid. Bad estimations do not. Usually scope creep is itself revealed when a new facet of the problem comes up, one which represents a competitive opportunity, which the client is happy to pay for.

Saying that no estimate should ever be good is like saying programmers should make gigantic mistakes. This is only possible when working on W-2, and only when the company won't die. End of discussion. Independents have to get it right. Any thoughts to the contrary will soon land an independent programmer back in W-2. The fact is that clients like predictable cost, and programmers who can deliver it can ask for more because they don't have to start squirming around in client meetings later and making justifications that are inevitably open to self-serving padding and other temptations. Be brave and keep scope, or go W-2.

Re: Why are software development estimates regularly off by a factor of 2-3?

#103
post #18

So the article tries to describe the hidden complexity of software with a hiking analogy but then it leaves a huge part missing. Let's extend the analogy somewhat... this isn't your first hiking trip. In fact, you've been on dozens and dozens of hiking trips. You're an experienced hiker. Yet, why do you continue to give meaningless as-the-crow-flies estimates of how long it will take you to get to your destination? T…

Let me fix that for you:

Developer estimates are regularly off because "experienced development managers" are all too eager and willing to allow organizational politics and external influences to add complexity in ways the developers cannot anticipate. Inexperienced developers' estimates are even further off because of their relative lack of experience.

I assure you, the first time a "development manager" tries to scapegoat his subordinates for his own failures to deliver to these external influences his subordinates realize that their estimates impact them eventually, and directly.

Re: Why are software development estimates regularly off by a factor of 2-3?

#104

Because management looks at you funny when you say something simple will take 4 months. Then they lie and say that it couldn't possibly take that long. And the reason they lie is because there's no negative to them tying to you. Their job is to get you to go faster, and they are allowed to lie as part of their job, if it makes you go faster. The reality is that it doesn't make people go faster. It makes people leave…

Management giving estimates? No thanks. I'd rather them take my input and do their job of understanding trends. When I say something takes a long time and it takes longer, I'm not bullshitting. When I think something takes a long time and doesn't, its generally because shortcuts are rarely understood up front. Will I find the same shortest path next time? If I'm doing the same exact work maybe. I do work in a place w…

If you think about it, though, estimates (time, money, personnel) is really a management function.

You write "I'd rather them take my input and do their job of understanding trends." that's exactly what I'm talking about. They need to take your input, and do their job.

If a worker is told: "Here is what you (personnel) have to do (task), with these tools (environment), and we want you to do it in this amount of time (money)", and the worker fails to complete the work in the amount of time, then it is management's fault. They improperly estimated the amount of time it would take the worker to perform his task.

That management ask the worker how long the task will take means management does not understand the task, and if management does not understand the task, how can they know whether the worker's skill, experience, and knowledge is adequate for the task? Now, they also seem to not understand the worker's skills, experience, and knowledge, so the problem is compounded.

It is a management function to clearly define the tasks that must be performed, attract people whose skills, experience, and knowledge fit the tasks, and fund these people with the appropriate environment, tools, and salaries for them to perform the tasks.

That is the role of management. If they are unable to estimate what tasks must be performed, what people need to be retained, and how much money and time will be spent, they fail at their managerial duties.

That they then blame the workers themselves for improperly estimating the work is doubly wrong, a sign of managerial immaturity.

Re: Why are software development estimates regularly off by a factor of 2-3?

#105
To put it simply: software estimates are hard because nobody has made that particular bit software before. Why? Because software doesn't rust. There is little need to exactly duplicate software which has already been written. Thus, most software is charting unknown territory.

Re: Why are software development estimates regularly off by a factor of 2-3?

#106
Dumbing down analogies aside, you can look at software development as a graph of connected tasks with each task having some variance around when it would complete. This variance accumulates as taskw are completed and you move on to next nodes in the graph. When you complete the graph, you have large probability distribution about average ETA.

Most decision makers have to choose some number for ETA however because world does not understand probability distributions or variances. Most likely this has to be lower end of the number to get project sponsored or fend off the competition.

This is not unique to software development. Any projects that requires large connected graph of small task nodes would show same characteristics. For example, fighter jets, Boeing Dreamliner, space station etc etc.

Re: Why are software development estimates regularly off by a factor of 2-3?

#107
I have found that providing estimates within a range is the most reasonable approach.

The range is usually between "this is the likely amount of time with everything going as it should" and "this is the worst case scenario time with all that can go wrong".

It's a pretty broad definition between the two but I find it gives clients a sense of the scope of the project. The minimum time does include some padding to include all the usual likely overruns (testing, requirements gathering, etc.) but not too much.

The worst case time better really be worst case as I provide clients the expectation that it should not take longer than this.

I find that usually I will end up somewhere in the middle of the estimate and the client is still happy since they didn't hit the worst case scenario.

I find that this usually tends to be relatively effective even if as a rule of thumb the worst case time is 3-4x the minimum time.

Of course, there is still a lot of analysis done to come up with this range and there is a lot of discussion with the client to help them understand the details on why and how it could go.

At the end of the day, communication is king. I find that educating clients about the complexities of the process always helps to gain trust.

Re: Why are software development estimates regularly off by a factor of 2-3?

#108
With 0.1% of all software representing 99.9% of all value ever produced in software, the question does not even particularly matter. Who the hell cares today how long it took or how much it cost to produce Google Search? Unless there is a good reason otherwise, the project that you are working on right now, has no economic value anyway. It is simple. If the budget or the duration matters, the project does not.

Re: Why are software development estimates regularly off by a factor of 2-3?

#109

Software-development estimates are regularly off by a large margin for two reasons. First, the problem is inherently hard. Second, social and business pressures bias the estimates downward. Why is the problem inherently hard? To see why, let's imagine that the All-Knowing Fairy Godmother of Software Estimation descends from the heavens and lends to us her magic estimating function F that, applied to any software proj…

For what it's worth, I haven't seen hardware engineers complaining about this issue as much as software engineers, which I find baffling. I think if you're going to justify why software development is such a "hard" problem, you should mention what makes it so different from other fields.

Re: Why are software development estimates regularly off by a factor of 2-3?

#110
People are generaly optimists and with that anything in the future will be biased towards ideals, even planning for exceptions will bias towards a ideal.

The other factor is feature creep, if you find a project so well defined and set in stone then it would be fair to say it will have a better chance of finishing on time.

That is just the software side, then there is the hardware to run it and that opens up a whole new aspect of poetntual delays that will get leveridged at the project and when people say project behind involving software people will just focus on the software aspect and all other factors are buried.

Bugs and issues come from many factors and even suppliers of hardware can have bugs which can have a impact, design could have bugs and many aspects can have issues that impact the end result.

But it is impossible and silly to plan for every situation as whatever you plan for there will be an exception and you get a deminishing return, hence easier to have a area of time labeled contingency as a catch all. Best example of such exceptions would be say an earthquake killing your entire development team, that can and has happened and at the same time is not something you plan for. Partialy why we have insurance. Though you will find it very hard to impossible to get insurace again software development delays beyond life/death of parties involved, though that will not chance the delay into a non-delay, mearly compensate.

With all this Engineers have for many lifetimes gone with the engineering factor or whatever you think it will take, muliply that by 2-3 to counter optimisim. But that does not help feture creep and anything added to the design after code has started is creep.

Post reply on HN