What do you mean ‘we need more time’? Project schedule estimation in software

One of the fundamental issues is that upper management sees estimates as bell curves as though 50% of the time a project may finish early. In reality, very few projects on time, let alone early. No one wants to hear about reducing scope but at the same time an MVP that solves the customer problem without feature creep should be evangelized within organizations. In reality, schedule follow more of an exponential function where slippage time goes from bad to worse very quickly as resources/scope/schedule snowball very quickly. Things get scrapped before we see this in practice.

For the kinds of projects my company tends to do, when we're getting into repeat business we're generally not competing with other contractors. Our clients know that the other contractors would have a ramp-up cost that we don't, and an unknown working relationship vs our good working relationship. Instead, they look for per-hour discounts and ways to 'cut costs' while delivering the same value. They also typically have their own development team who don't get paid nearly as much per-hour as my company charges, so they look to share more of the work with their internal team. Since we do product, services, and support, including training, it's hard to argue that the internal team can't or shouldn't do the work.

It gets complicated, but in the end we mostly come out alright. Our clients are well aware that taking on additional work reduces our likely development cost but increases overhead costs and raises project/schedule risks. It's a tradeoff they accept, and we work with them to ensure the project is successful.

My point is, in the real world you can't just charge by the value you provide rather than by the cost of providing that value. That's a simplistic point of view.

The game is rigged. Only way to win is not to play.

As a general rule, if management expects results without giving you the tools for the job, ask yourself whether you really want to be in that job, and start looking for alternatives. That said, there's several ways you can make things better:

> What do you do when you don't have a spec and when you ask the senior engineer you are working with questions, they don't respond or push back against having a spec or a plan longer than the very next feature?

Illustrate why you need the answer. "I'm trying to decide between approach A or approach B. If we're doing X then approach A is easier to implement, but that'll suck if we're doing Y, in which case the extra work involved in B is worthwhile".

> The author says that estimation is easier to do if you are doing something you have done before. What do you do if you are always working with a new framework or language?

If you have experience with Express in Node.js, Ruby + Sinatra or Python + Flask will seem familiar to you. Rails or Django not so much. C++ would be more alien to you than Python or Ruby. Learn to identify such similarities and differences. I tend to find that domain knowledge is harder to come by than language/library knowledge -- The exact details of the work might be different, but if you're used to writing web apps or compilers or games or whatever, then you should be able to ballpark estimate such a project in a different language. Larger projects are easier too -- the cost for picking up the tools is roughly fixed and gets amortised over the whole lifetime of the project.

> - How do you break down the steps if you don't know what a project entails until you are finished?

First off: If we're talking about more than a couple of days' worth of work, estimating a project isn't a 5-minute job. Think a few hours for a 2-3 week project, at least, if you want a precise estimate. And there will be things you miss, but that's why you always add some padding.

At a minimum, you need to ask yourself the following questions: You're writing a service that does X. How does it receive input from the users? How does it pass output to the users? How does it obtain the data necessary to do X? Once you have the user input and the data, what work is involved in performing the task X?

Note how all these questions all apply irrespectively of whether you're writing a web service, a compiler or a game (with the details of what they _mean_ being quite different). You spend some time researching potential answers, and come up with a sketch of what an implementation looks like. Then you iterate: for each piece of the puzzle, what's involved. One of the easiest rookie mistakes to make here is to account for the time necessary to _implement_ all the pieces (and often that will be accurate), but allow remarkably little to no time for integrating the pieces together, for testing, for documentation, and for all other such things.

> What do you do if asked to estimate how long something will take to debug?

Tell them to f* off, in the most abrasive tone possible that is appropriate for your relationship with the other party. Seriously though, estimating how long it takes to sort out a bug is much, much harder than estimating new development. Time-to-fix-a-bug is also a long tail sort of phenomenon, so be prepared for that.

> This excercise takes a lot of time. What do you do when asked for an estimate in a meeting rather than over email?

After some projects you'll soon start being able to come up with a few good questions that need answering before you can give a good estimate. Bring them up, say you need to think about it, and be firm.

> - What do you do if the thing you are building relies on getting an external API to work and that API is either undocumented or is documented in a foreign language? Assume that the google translation is not making sense and this is your first project and you don't have a budget to hire a professional translator? This sounds like something you would need to get working before you could actually give an estimate.

Estimate the estimation. Say you'll need some time to figure out the requirements before you can safely say what work is involved.

> Why do experienced engineers ask for estimates anyway? Every time I give one I feel like I am lying and I try to warn people "I really don't know how to come up with estimates but I am guessing 3 hours". How do I deal with it when they are upset about it taking a week and a half?

This is probably the single most important part, and most places I've seen fuck it up tremendously. It's crucially important that people get to talk about this without turning it into a blame game. Why did you think it would take three hours, and why did it take a week and a half instead? Clearly either the estimation was wrong, or something _really_ bizarre happened that made the estimate invalid. Estimation is a skill that you need to learn, and you'll need to learn it before you're any good at it. My experience is that you _can_ estimate things correctly (down to the half-day on projects lasting multiple weeks), once you have the context to do it in.

For a lot of these, rather than trying to estimate the thing you're asked to estimate, you need to either estimate how long it'll take you to do the analysis you need in order to come up with an estimate, or you need to provide a time-limit on the task after which you'll provide a status update and have a discussion about the next step.

Managers don't really care about time, they care about budgets and risks. In your debugging example, the existence of the bug is a big unknown risk, and your manager needs to manage that. You obviously can't tell how long it'll take to fix the bug, but you can say "Give me an hour to debug it, and after that I should have a better idea of what the cause is and how long it'll take to fix. I might even have it fixed already by that point." That gives your manager a budget (one hour) and risk control (more information in a fixed time), and the option after a small investment to bring someone else in to assist.