How to replace estimations and guesses with a Monte Carlo simulation • Hacker News Board

Re: How to replace estimations and guesses with a Monte Carlo simulation

#131

Post by hinkley » Wed, Oct 06, 2021, 6:17 PM UTC

mgkimsal wrote: post #112

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

> I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Apparently I've worked with a lot of bozos over the last 25 years... (apologies to everyone I've worked with).

Lion tamer reporting in. It’s all circuses out there.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#132

Post by cratermoon » Wed, Oct 06, 2021, 6:22 PM UTC

I feel like the author just re-invented the XP concept of Velocity[1], but gave it a mathematical form.

1. https://martinfowler.com/bliki/XpVelocity.html "You should usually determine velocity by measuring how much got done in past periods, following the principle of YesterdaysWeather. A typical approach is to average the velocity the past three time periods to determine velocity for future time periods."

Re: How to replace estimations and guesses with a Monte Carlo simulation

#133

Post by alangibson » Wed, Oct 06, 2021, 6:25 PM UTC

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

> I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date."

I gotta tell you, you've never worked with a business stakeholder, the government (see financial penalties for missing deadlines) or with an absolute deadline (ie Christmas) if you think that's true.

> Many commenters on this story believe the main purpose of estimation is to assert when something will be done

We know our estimates can never be guarantees, but that is irrelevant to stakeholders whose career advancement depends on project X being done in day Y. Guess who's opinion carries more weight in literally every profit making enterprise.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#134

Post by cratermoon » Wed, Oct 06, 2021, 6:25 PM UTC

dagw wrote: post #114

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Happens all the time, and not just by bozos. Sometimes getting paid is entirely dependent on delivering when you said you would. I used to work at an animation studio and one year we worked on a series of Christmas themed TV spots. When we said we…

But they are bozos if they give you a task they want done for Christmas, you tell them that it will likely not complete until February, and they say, "well, just work harder/extra hours/hire more people", which is pretty common.

The way to avoid bozo-land is for the person asking to come back with "well, what can you confidently complete in time for Christmas?", and plan for that.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#135

Post by alangibson » Wed, Oct 06, 2021, 6:35 PM UTC

I tried this at a past job. I fed Jira history into a Monte Carlo simulation that would give a set of probabilities for various completion timeframes for any new user story. Despite months of trying to sell the idea, backed by sound mathematics instead of guesses, I never got past the point where stakeholders would listen politely then immediately go back to demanding to know "when will it be done"

Re: How to replace estimations and guesses with a Monte Carlo simulation

#136

Post by codemac » Wed, Oct 06, 2021, 6:44 PM UTC

sillysaurusx wrote: post #19

codemac wrote: post #13

Earlier quoted context omitted.

What happens when you complete your work before you know what you need to do next? If this never happens, then you have some invisible queue, as you do have things to do next. As far as your example, that's a great example of a task that seemed like it would take long, and ended up being very very short. Can you describe why this would be bad to add into your task system? - Add Task: Run banking API in $BigBank test…

Switching to prod mode takes 5 to 7 business days, because we have to order certs from DigiCert and then upload them to $BigBank, whose team requires 5 to 7 business days to activate said certs. We expected to turn on prod once testing was finished. But we ended up discovering that prod was the only correct test environment, because their test environment is rand() and fork()ed to the point that it doesn't even sligh…

Maybe I'm missing your point. It seems you're attempting to answer the wrong question: is this task accurate, given all the changes that have happened? This is irrelevant for large scale estimation.

The question for scheduling prediction is: what distribution of time will it take to mark any task in this queue as FIXED/INVALID/WONTFIX/OBSOLETE/etc? The queue can have any amount of vagueness you want in it.

Regardless of the embedded work, regardless of whether or not it changes, becomes invalid, doesn't exist, etc - these are all probability weights for any given task/project.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#137

Post by mateo411 » Wed, Oct 06, 2021, 6:56 PM UTC

game_the0ry wrote: post #80

myth2018 wrote: post #74

I think it's understandable why programmers dread doing estimates so much. But I believe there are multiple factors contributing to the current overall mess and the guilt should be shared among several areas -- and that includes us. Estimates are often, if not always, mistaken as commitments or deadlines. But we don't do a great job on explaining that estimates are probabilistic by nature, there are risks involved, a…

> Estimates are often, if not always, mistaken as commitments or deadlines This. This is the problem with "estimates." My solution - and this will sound bad - is to over-estimate.

Yes, this is everyone's solution, all the way up the organizational hierarchy.

Software Engineer, it will take me 2 weeks. Engineering Manager it will take us 4 weeks. Product Manager it will take them 8 weeks. VP it will be done by the end of next quarter.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#138

Post by xyzelement » Wed, Oct 06, 2021, 7:17 PM UTC

alangibson wrote: post #133

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

> I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." I gotta tell you, you've never worked with a business stakeholder, the government (see financial penalties for missing deadlines) or with an absolute deadline (ie Christmas) if you think that's true. > Many commenters on this story believe the ma…

You and a few other responses hit on a related but distinct problem: an unmovable deadline.

In this case, the question is no longer "how long will X take" but "but what will it take for X to complete by Y." That can require aggressively cutting scope, reassigning resources, de-risking in every possible way. That's quite different than asking a developer/team "how long this will take" assuming constant resources and given scope.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#139

Post by cratermoon » Wed, Oct 06, 2021, 7:31 PM UTC

mgkimsal wrote: post #125

Mtinie wrote: post #86

Earlier quoted context omitted.

“Don’t worry, this is _purely_ an estimate and would never be used to set the deadline. So just give us something to share with the ELT…”

"Just take a guess - we have to have a number in this box by the end of the meeting". I was literally just told that yesterday. No one else who would look at that number will know it was a totally random guess pulled out of thin air to satisfy ceremonial box-filling. They will treat it as an estimate, and make plans accordingly.

And if you make up a number that is too obviously small or exceeds some unspoken upper bound, you'll be asked to re-estimate anyway. Sometimes the best way to respond to that is to finesse the discussion into coming up with a number that, while it will have no relationship to the actual effort, at least reflects what the stakeholders hope and desire it to be. At that point the team, if they are smart, will examine scope and re-plan to come up with some level of effort that they are confident can be done in the time hoped for.

In other words, get the people who want a number to tell you what number they want, then use your best efforts to scope the effort to one you can be pretty confident will fit.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#140

Post by digikata » Wed, Oct 06, 2021, 8:00 PM UTC

alangibson wrote: post #133

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

> I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." I gotta tell you, you've never worked with a business stakeholder, the government (see financial penalties for missing deadlines) or with an absolute deadline (ie Christmas) if you think that's true. > Many commenters on this story believe the ma…

There is a reason project management calls scope-cost-time the iron triangle. It's extremely to bind all three sides tightly unless the estimates are very good, and/or there is significant added buffer to the project (and that really means the cost side is what's stretching to give you extra certainty on the hard scope and time deadlines).

More commonly at most two of the three are tightly constrained, and another side is more free. One can optimize in some directions, but none of the sides are fully independent of each other. I think the best project outcomes I've seen on are ones where only one side is strongly fixed, and the team is in control of the other two aspects.