Live data from Hacker News

Why software projects take longer than you think – a statistical model

erikbern.com

61–70 of 324 posts

Re: Why software projects take longer than you think – a statistical model

#61

An important aspect of being a professional software engineer is having the backbone to sometimes say things like: - “I don’t know yet enough about the problem to give you even a rough estimate. If you’d like, I can take a day to dig into it and then report back.” - “This first part should take 2-3 days. 5 on the outside. But the second part relies heavily on an API whose documentation and error messages are in Chine…

I once had a manager/CEO who thought you can negotiate the planning, the same as like a price. "The customer always wants to get it earlier, the engineer later". Me saying that only the developer is able to give a realistic planning, was probably received as "developer wants to push the deadline back in his negotiation". In the end, his company went bust because he was selling stuff to customers that could never be d…

There is a type of negotiation that can be done: Negotiation about scope. Work doesn’t get compressed. But stakeholders can identify non-essential business-facing features and cut scope.

Note well: Automated testing is almost certainly essential. If someone says you are “building an MVP”, then make sure it is an actual MVP. Make sure you have a clear market hypothesis to test and check that it couldn’t be built out of duct tape, Google Forms, Zapier, ITTT, manual processes, and excel-wrangling.

Re: Why software projects take longer than you think – a statistical model

#62
I’ve been coding professionally for 20 years and I’m largely no better at estimating completion than day 1, maybe worse since I’m more likely to feign confidence in a figure I’ve essentially pulled out of my ass.

At some point, I came to the realization that my fundamentally poor concept of time was always going to be an insurmountable obstacle to my career advancement.

Re: Why software projects take longer than you think – a statistical model

#63

An important aspect of being a professional software engineer is having the backbone to sometimes say things like: - “I don’t know yet enough about the problem to give you even a rough estimate. If you’d like, I can take a day to dig into it and then report back.” - “This first part should take 2-3 days. 5 on the outside. But the second part relies heavily on an API whose documentation and error messages are in Chine…

I would recommend this talk regarding professionalism:

https://www.youtube.com/watch?v=p0O1VVqRSK0#t=5m20s

EDIT: The part about estimates is here: https://www.youtube.com/watch?v=p0O1VVqRSK0#t=36m56s

Re: Why software projects take longer than you think – a statistical model

#64
Well, I think this is a complex topic. Not because of the math, but because the key to an accurate estimate is to understand who has done the estimate and on what basis.

As stated in the summary, the core driver for inaccurate estimates is uncertainty:

> Tasks with the most uncertainty (rather the biggest size) can often dominate the mean time it takes to complete all tasks.

There are different sources of certainty:

- Experience: If someone has done a task 20 times he probably knows how much time he will require. Someone who hasn't done the task yet, probably underestimate the time he requires (e.g. because of the median vs. mean conflict). But don't be fooled just because you have 20 years of work experience in the field but never actually done a specific task doesn't mean you can estimate it better than someone who just started the job but had done that specific task 10 times. However, most of the time, projects are doing something new. So you have to find out which tasks have been done before by a project member and which are completely new ground. If something is completely new, remember to plan time for getting familiar with a problem space plus a handful of complications (together this will be more than the actual task would take someone who is trained for that specific task).

- Detail: The smaller the tasks the larger the overall estimate... or so. Planning on a top-level is rarely going to be accurate. We do it a lot because it doesn't take much time. But if you want an accurate estimate, you have to plan on small, specific tasks.

- Risk management: Every project has risks. Some don't really have an impact and others blow up the whole project. Know your risks and what you are going to do if something should go in the wrong direction. It is not like you wouldn't have time to figure out what to do when the problem occurs but to understand how it would impact your timing and to take preventive actions (e.g. include stakeholders).

If you have people who have done the exact same task a few times, made a detailed plan of every step and know how to handle the most likely or impactful risks you are in a good position to deliver on time. Most of the time you won't have that luxury and have to compensate the resulting uncertainty with a prolonged time to project completion, but that should be just fine when being communicated in the beginning.

With all that said, remember, that some projects don't require an accurate estimate. Sometimes it is enough to deliver just as soon as possible.

Re: Why software projects take longer than you think – a statistical model

#65

Earlier quoted context omitted.

The above approach uses the factor 2.2. Why would your factors be any better?

Wouldn't it be 2.4 as you are adding 20% of the already doubled estimate?

Yes.

Re: Why software projects take longer than you think – a statistical model

#66

An important aspect of being a professional software engineer is having the backbone to sometimes say things like: - “I don’t know yet enough about the problem to give you even a rough estimate. If you’d like, I can take a day to dig into it and then report back.” - “This first part should take 2-3 days. 5 on the outside. But the second part relies heavily on an API whose documentation and error messages are in Chine…

On the other hand, you can only argue about the estimate if there is even a channel to argue about it. I have seen several cases of (1) people being told about a deadline through various proxies, so there was no point in time where they could even question the imposed deadline, (2) people not being told about the deadline at all until it was missed, so they could not argue against it, (3) people not being told about the task that has a deadline until after that deadline was missed.

All involved engineers actually had the backbone to say it was impossible, but no chance to say it. Obviously, projects did not go well at that place.

Re: Why software projects take longer than you think – a statistical model

#67
post #63

An important aspect of being a professional software engineer is having the backbone to sometimes say things like: - “I don’t know yet enough about the problem to give you even a rough estimate. If you’d like, I can take a day to dig into it and then report back.” - “This first part should take 2-3 days. 5 on the outside. But the second part relies heavily on an API whose documentation and error messages are in Chine…

I would recommend this talk regarding professionalism: https://www.youtube.com/watch?v=p0O1VVqRSK0#t=5m20s EDIT: The part about estimates is here: https://www.youtube.com/watch?v=p0O1VVqRSK0#t=36m56s

I'd replace "they are hired for their knowledge" with "professionals are hired for their judgement". Judgement usually depends upon knowledge, but its the exercise of judgement that is more key to emphasise.

Re: Why software projects take longer than you think – a statistical model

#68
post #39
post #28

Earlier quoted context omitted.

there is an uncharitable response to this thread, which is marked dead. I do think there is a kernel of truth in that response, which is nothing more than mocking. why fibonacci? I think it's reasonable to say time includes an exponential error. an estimated 1 hour task is very different than an estimated 1 week task. i see that 1 hour, 1 day and 1 week estimates are progressively, and likely exponentially, worse. Is…

The kernel of truth of that response is that you won't get more certainty about an estimate by adding more uncertainty, no matter in how much pseudo-scientific cargo-culting mumbo-jumbo you cloak it. (Unfortunately, HN readers don't really like sarcastic commentary which requires reader to think it through. Time is precious, and we just want to be coddled, give me your opinion ELI8 straight, because there is no time…

I think you're half right. adding an error band to a far out estimate won't give you a better estimate, but it might help tell you how much you _don't know_ about what you're facing.

Cumulative error is _hard_. see weather prediction.

Re: Why software projects take longer than you think – a statistical model

#70
I think the most important issue why projects take longer is not because the time it takes to complete a task is uncertain, but because at the start it will always be unknown which tasks will prove critical. The further you get, the more tasks will reveal itself that were not part of the original scope, but critical nonetheless.
Post reply on HN