Live data from Hacker News

How to replace estimations and guesses with a Monte Carlo simulation

lucasfcosta.com

91–100 of 162 posts

Re: How to replace estimations and guesses with a Monte Carlo simulation

#91

A more “back of the envelope“ approach to handling uncertainty for this sort of thing is the “three point estimate” approach, where you give a best, worst, and most likely estimate for each sub task. These points are implicitly used to parametrise a distribution, and then you analytically find the overall uncertainty, rather than through simulation.

We used a different three point system. 1 point means we know exactly how to accomplish this work. 2 means there is uncertainty, but we are confident it will completed. 3 means there are unknowns that require more digging. A 3 pointer is a good candidate for breaking the story/task up into smaller chunks.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#92
post #39

Earlier quoted context omitted.

> A very nice thing about Monte Carlo simulation is that at the end your distribution of results are all within the feasible range. I guess Monte Carlo helps provide conservative estimates behind a façade of rigour, but the truth of the matter is that in the end it's still GIGO. Any empirical distribution only reflects the empirical measures that were used to generate it. If you bundle everything from the time it too…

I’m smiling a bit while reading your comment. You are completely correct that that estimate is virtually meaningless. And yet… that doesn’t mean it’s completely useless! I’ve been using a somewhat meaningless Monto Carlo inspired approach to project planning for a while in my consulting company. The vast majority of the projects I do have only a small amount in common with previous projects, so when I’m estimating I’…

The traditional distribution for costs and dates is Beta-PERT. Douglas W. Hubbard uses log-normal distribution. I am a pessimist, so I like Beta-PERT with a fat, fat tail.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#93
post #88

You don't need a monte carlo simulation if you have a known distribution. You can just add the means and variances and solve the equation rather than approximating it with a simulation. Monte carlo is more useful when drawing from an unknown distribution.

My understanding is that while the normal distribution is friendly to simple manipulation, a lot of distributions aren't. Not being a statistician I couldn't tell you why or which.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#94
post #87

I've actually been working on turning this idea into a product for the past year or two. Looks like I'm not the only one to have thought of this!

I worked on something similar a while ago. I was defeated by UX more than the mathematics.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#95
Monte Carlo simulations are invaluable when it comes to creating tissue line odds. For one of my horse racing projects, I can come up with a range of how an individual horse is likely to perform-- which isn't much use unless compared against the others in the race with a frequency component (e.g., this horse will likely be the best only 10% of the time-- but the pools are offering 15-1 odds, so it is +EV). On modest hardware, I can run thousands of race simulations and somewhat effectively determine the likelihood of a given horse winning.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#96

I recommend the book "Software Estimation: Demystifying the Black Art" (2006) by Steve McConnell. Nice collection of excerpts: https://leventov.medium.com/excerpts-from-software-estimatio... There is also a short chapter on estimation tools using e.g. Monte Carlo simulations. One tip which is relevant here and that is also more or less presented in the article and probably obvious :) "TIP #65: Don't treat the output…

Some other books I can recommend to read alongside McConnell's are Waltzing With Bears by DeMarco and Lister (of Peopleware fame) and How to Measure Anything by Douglas W. Hubbard.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#97
post #43

While a different application than shown here, my project management class taught me one of the biggest benefits of Monte Carlo simulation - estimating uncertainty. Traditionally, net present value calculations are done with single point estimates. For example, analyzing a rental property we want to buy, we'd estimate the vacancy rate, interest rate, property appreciation, maintenance expenses, and do all of that on…

One of the concepts that many people don’t get is differentiation between risk and uncertainty. These same people will then attempt to “model risk” and end up with (ironically) an “uncertain estimate” https://en.m.wikipedia.org/wiki/Knightian_uncertainty

They define risk as having statistical noise or a varying parameter in the estimate, while uncertainty is lacking information about the target. Doesn't the presence of noise in an estimate imply that there's information you aren't accounting for? I don't see the importance of the distinction.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#98
Speaking purely to Monte Carlo simulations: one key issue is knowing the underlying distributions of your random variables. Without this, you may be led further astray than before your simulation.

A much more robust approach - though not always appropriate - is a worst-case vs middle vs best-case scenario analysis

Re: How to replace estimations and guesses with a Monte Carlo simulation

#99

Earlier quoted context omitted.

Yeah, FogBugz had a very similar feature 15 years ago. It was a hybrid between the approach proposed in the article, and the "always multiply by N" technique. Basically, FogBugz always asked you for your own best estimate. It tracked how much longer the actual time to completion was compared to your best estimate, and learned a simplified statistical distribution of your personal multipliers (e.g. in 80% of cases you…

I never tried FogBugz but I heard good things about it, similar to what you've written. It's surprising it didn't catch on more if it's that good? I mean, I suppose it could still catch on, the product still exists as far as I know.

I’ve written about why FogBugz failed before, but it basically came down to Atlassian having vastly better marketing, being much more willing to support every edge case instead of being opinionated, and having a better pricing model.

Unrelated, the new owners of Kiln and FogBugz are asshats, I moved my stuff off ages ago, and I’d encourage anyone else still on them to do the same.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#100
Whilst I'm happy that probabilistic methods of estimation get air time, I feel these sorts of posts give the lay person unearned confidence that they are suddenly far safer then they are. In most cases I find people don't know what should be planned, how it should be defined, measured, tracked or assured - and when combined with a lack of appreciation for Risk and how it should equally be defined, measured, tracked and controlled this is the core problem not idea of estimating itself.

### Skip to TLDR ###

Usually it's of little consequence and the stakes are low, but it's this general lack of skill in such a large population of people who are doing it poorly that gives estimation such a bad reputation in the first place. The idea of estimation is not incorrect, it's that people aren't taught how to do it properly in the first place and industry continues to propagate the myth that it's simpler than it is. Often because it's mandated by those who don't understand it themselves.

Having played in many types of industries I have a thesis that much of what we do in planning and estimating has had real benefit in some areas historically - where there are clear, repeatable, measurable observations, certain types of construction and fabrication - but not others due to their success being tied to the method alone without account for how the nature of what is being measured.

Monte Carlo has very real and valuable uses in critical path method scheduling, but, as with most things, only when used appropriately. Weather is something that fits well, because it is cyclic, we have a lot of data, we can, ahead of time, to some degree, have a likely view of where and when things like hurricanes/cyclones will occur in a given period.

In the Australian North-West Shelf where I used to plan O&G maintenance dive campaigns, knowing how long it takes to get to an area, a diver to get to depth, perform a task, return safely and leave the area is the easy part. "Everything going to plan"* we should be able to do these x number of inspections/interventions in y number of days. But weather is where you can come unstuck. Sea state matters, there is always some chance that you will have to interrupt or postpone on a day because of conditions. We know there are roughly x number of these days per cyclone season and roughly how long we might not be able to work for but pay full rates for crew, equipment etc.. An at risk period for weather goes on to the end of the schedule as contingency and is drawn down as needed, but in some scenarios where in the network of activities these multi-day events occur can have varying impacts. Inserting a three day stand-down early on could have a greater impact on the end date than a three day stand-down right near the end if at the earlier time you are doing things that must be done in a continuous sequence with - if interrupted - will need you to repeat again from the beginning. Luckily there are tools we can use to not just look at the net effects of activities in a schedule network having their own distribution curves, but also injecting events, again with their own probabilities, randomly throughout the network during the run.

This gives you a much greater insight than a 'P90' estimate on your critical path, it gives you the sensitivity of activities in your network.

### TLDR ###

This all assumes, of course, that what you have modelled (planned) reflects reality to some degree. A bad schedule with bad bounds will give you a bad result regardless. It doesn't matter how many times you simulate it.

*it rarely does

Post reply on HN