Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to replace estimations and guesses with a Monte Carlo simulation

lucasfcosta.com

Post reply on HN
Print view View original

11–20 of 162 posts

Re: How to replace estimations and guesses with a Monte Carlo simulation

#11

Post by cycomanic »

asdf_snar wrote: post #4

I really dislike posts in this vein, because they use statements like "This post will teach you how to replace estimations and guesses with a Monte Carlo simulation." as though the output of a Monte Carlo simulation were not an estimate, but some higher truth. The process of simulating many different outcomes with a touch of randomness gives the layperson the impression that we are really _doing_ something, when in f…

I sort of agree with you although for different reasons. The problem with the MC approach IMO is not really that it is also an estimate, estimates are fine. However doing a Monte Carlo simulation is only reasonable if we have a good model. So now we have moved from we can't estimate (intuitively I guess) to how to find a good model, that is not trivial and the time spend on finding and verifying the model might take more time than what you are actually trying to do (just an (gu-)estimate on my part)

Re: How to replace estimations and guesses with a Monte Carlo simulation

#12

Post by Ntrails »

DavidPeiffer wrote: post #5

While a different application than shown here, my project management class taught me one of the biggest benefits of Monte Carlo simulation - estimating uncertainty. Traditionally, net present value calculations are done with single point estimates. For example, analyzing a rental property we want to buy, we'd estimate the vacancy rate, interest rate, property appreciation, maintenance expenses, and do all of that on…

The real danger is not understanding that this "uncertainty" estimate is a function of your assumptions. How you model the distribution of your inputs is huge, and often not stated clearly.

GIGO

Re: How to replace estimations and guesses with a Monte Carlo simulation

#13

Post by codemac »

sillysaurusx wrote: post #7
brightball wrote: post #2

I’m a big fan of the Donald Reinertsen approach: measure queue length. Simply track the time to complete each task in the team queue on average, then multiply that by the number of tasks remaining in the queue. Each team will habitually slice things into sizes they feel are appropriate. Rather than investing time to try and fail at accurately estimating each one, simply update your average every time a task is comple…

What do you do when your future tasks are unknown or ambiguous? For example, at my day job my task is to implement banking. The day to day tasks change... day to day. There aren't a "number of tasks remaining in the queue," since whatever I'm doing is what I'm doing. One could say this is poor planning. But due to the nature of Big Banks, each task is usually blocking the next one -- in other words, it's not possible…

What happens when you complete your work before you know what you need to do next?

If this never happens, then you have some invisible queue, as you do have things to do next.

As far as your example, that's a great example of a task that seemed like it would take long, and ended up being very very short. Can you describe why this would be bad to add into your task system?

    - Add Task: Run banking API in $BigBank test environment
    - Start work time clock.
    - Find out we don't need to do it, and switch to prod mode
    - switch to prod mode
    - Close task, and time clock
This is now data for your estimates of future tasks, as this will probably happen randomly from time to time in the future.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#14

Post by xornox »

After many decades we still discuss and wonder how work amount estimates should be done and why they are always wrong. The most brilliant software engineers and project managers have developed countless different methods and nothing works.

For me it tells that nature of work is such that we just can not estimate work amounts. We are just pretending that it can be done.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#15

Post by edu-ap »

asdf_snar wrote: post #4

I really dislike posts in this vein, because they use statements like "This post will teach you how to replace estimations and guesses with a Monte Carlo simulation." as though the output of a Monte Carlo simulation were not an estimate, but some higher truth. The process of simulating many different outcomes with a touch of randomness gives the layperson the impression that we are really _doing_ something, when in f…

A model is not required if you use data from your team's past performance with a tool like this: https://marketplace.atlassian.com/apps/1216661/actionableagi...

Re: How to replace estimations and guesses with a Monte Carlo simulation

#17

Post by mach1ne »

xornox wrote: post #14

After many decades we still discuss and wonder how work amount estimates should be done and why they are always wrong. The most brilliant software engineers and project managers have developed countless different methods and nothing works. For me it tells that nature of work is such that we just can not estimate work amounts. We are just pretending that it can be done.

It depends on the type of work. Generally and counterintuitively, the more individual random variables the work contains the easier it is to estimate. This is because random variations cancel each other out when they are in large quantities.

With programming, one random variable can add days, weeks or even months of work.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#18

Post by Loic »

Ntrails wrote: post #12
DavidPeiffer wrote: post #5

While a different application than shown here, my project management class taught me one of the biggest benefits of Monte Carlo simulation - estimating uncertainty. Traditionally, net present value calculations are done with single point estimates. For example, analyzing a rental property we want to buy, we'd estimate the vacancy rate, interest rate, property appreciation, maintenance expenses, and do all of that on…

The real danger is not understanding that this "uncertainty" estimate is a function of your assumptions. How you model the distribution of your inputs is huge, and often not stated clearly. GIGO

GIGO, the first thing I learnt, as I entered the industry 20 years ago. This was from a 60 year old engineer who told me that experience is only a nice name for "all the @#$% I made I will try not to make again".

A very nice thing about Monte Carlo simulation is that at the end your distribution of results are all within the feasible range. If you do error propagation using uncertainty on your parameters, you can get non-sense results.

For example, suppose you have a pendulum, you simulate it with error propagation and you have a non 0 probability of having an increase of the energy in the system over time.

It is easy to spot this in such constrained example but with more complex models, this is sometimes pretty hard to control.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#19

Post by sillysaurusx »

codemac wrote: post #13
sillysaurusx wrote: post #7

Earlier quoted context omitted.

What do you do when your future tasks are unknown or ambiguous? For example, at my day job my task is to implement banking. The day to day tasks change... day to day. There aren't a "number of tasks remaining in the queue," since whatever I'm doing is what I'm doing. One could say this is poor planning. But due to the nature of Big Banks, each task is usually blocking the next one -- in other words, it's not possible…

What happens when you complete your work before you know what you need to do next? If this never happens, then you have some invisible queue, as you do have things to do next. As far as your example, that's a great example of a task that seemed like it would take long, and ended up being very very short. Can you describe why this would be bad to add into your task system? - Add Task: Run banking API in $BigBank test…

Switching to prod mode takes 5 to 7 business days, because we have to order certs from DigiCert and then upload them to $BigBank, whose team requires 5 to 7 business days to activate said certs.

We expected to turn on prod once testing was finished. But we ended up discovering that prod was the only correct test environment, because their test environment is rand() and fork()ed to the point that it doesn't even slightly resemble the prod environment. Hence, "prod am become test, destroyer of estimates."

So for 5 to 7 business days, we'll be building out our APIs by "assuming a spherical cow," i.e. assuming that all the test environment brokenness is actually working correctly (mocking their broken responses with non-broken responses.) Then in 5 to 7 business days, hopefully we'll discover that our spherical-cow representation is actually closer to the physical cow of the real production environment. Or it'll be a spherical cow and I'll be reshaping it into a normal cow.

By the way, if you've never had the pleasure of working with a $BigBank like Scottrade, Thomson Reuters, or $BigBank, let's just say it's ... revealing.

Post reply on HN