How to replace estimations and guesses with a Monte Carlo simulation • Hacker News Board

Re: How to replace estimations and guesses with a Monte Carlo simulation

#41

Post by conjectures » Wed, Oct 06, 2021, 9:29 AM UTC

Loic wrote: post #18

Ntrails wrote: post #12

Earlier quoted context omitted.

The real danger is not understanding that this "uncertainty" estimate is a function of your assumptions. How you model the distribution of your inputs is huge, and often not stated clearly. GIGO

GIGO, the first thing I learnt, as I entered the industry 20 years ago. This was from a 60 year old engineer who told me that experience is only a nice name for "all the @#$% I made I will try not to make again". A very nice thing about Monte Carlo simulation is that at the end your distribution of results are all within the feasible range. If you do error propagation using uncertainty on your parameters, you can get…

> If you do error propagation using uncertainty on your parameters, you can get non-sense results.

Only if you do it wrong, generally. Events which are impossible under some hypothesis should have zero probability under a model for it and not be sampled.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#42

Post by tchalla » Wed, Oct 06, 2021, 9:43 AM UTC

alfonsodev wrote: post #31

Not to hijack the conversation, but I though some people interested in replacing estimations could find this useful. In order to replace estimation, we are trying Basecamp's Shapeup language and techniques [1] In a nutshell the thinking is reversed, instead of asking: - "how long would it take to implement X?" you ask: - "how much appetite (in weeks and people involved) do I have for this feature ?" Then you work wit…

Flipping the question within a restricted variable is my favorite hack.

Overlords - “How much money do you need for this project?”

Me - “How much money can we spend?” / “How much time should we spend?”

Friend- “Look at how much money that CEO earns!”

Me - “What will you give up in your lifestyle to do what the CEO does?”

The only challenge though in these situations is that most people aren’t open enough to do “Think in bets”. They either avoid the question or don’t even attempt to come an answer (despite them not spending a single dime!).

Re: How to replace estimations and guesses with a Monte Carlo simulation

#43

Post by tchalla » Wed, Oct 06, 2021, 9:46 AM UTC

DavidPeiffer wrote: post #5

While a different application than shown here, my project management class taught me one of the biggest benefits of Monte Carlo simulation - estimating uncertainty. Traditionally, net present value calculations are done with single point estimates. For example, analyzing a rental property we want to buy, we'd estimate the vacancy rate, interest rate, property appreciation, maintenance expenses, and do all of that on…

One of the concepts that many people don’t get is differentiation between risk and uncertainty. These same people will then attempt to “model risk” and end up with (ironically) an “uncertain estimate”

https://en.m.wikipedia.org/wiki/Knightian_uncertainty

Re: How to replace estimations and guesses with a Monte Carlo simulation

#44

Post by m12k » Wed, Oct 06, 2021, 9:47 AM UTC

DavidPeiffer wrote: post #5

While a different application than shown here, my project management class taught me one of the biggest benefits of Monte Carlo simulation - estimating uncertainty. Traditionally, net present value calculations are done with single point estimates. For example, analyzing a rental property we want to buy, we'd estimate the vacancy rate, interest rate, property appreciation, maintenance expenses, and do all of that on…

Estimating uncertainty is preferred, but if you have people who insist on a single number, then teach them that it should be the median instead of the mean. Means tend to overvalue "moonshots" where there's a 99% chance you lose money, but if the payoff in that 1% where you win is large enough, it can still result in a positive mean. And a mean can often be an outcome that isn't actually possible, but falls between several options, while a median is always an outcome that can actually happen, and there's ~50% chance you'll get a better one.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#45

Post by ImaCake » Wed, Oct 06, 2021, 9:50 AM UTC

BoppreH wrote: post #23

I love small simulations like this, but I find tiresome writing all the scaffolding to summarize the results. So I wrote this Python module: https://github.com/boppreh/carlo pip install carlo carlo "d(20)+randint(2, 5)+int(random()*20)" This will continually generate samples and show them in a self-updating histogram. This is harder than it looks, because there's too many samples to store them all, and we don't know…

That is pretty neat! I have a dice stats python script I use specifically for DnD. It has command line flags and basically the same syntax as the discord dice bot. Very useful when you are considering strategy, even though my group plays fast and loose with the DnD rules.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#46

Post by ghiculescu » Wed, Oct 06, 2021, 9:51 AM UTC

alfonsodev wrote: post #31

Not to hijack the conversation, but I though some people interested in replacing estimations could find this useful. In order to replace estimation, we are trying Basecamp's Shapeup language and techniques [1] In a nutshell the thinking is reversed, instead of asking: - "how long would it take to implement X?" you ask: - "how much appetite (in weeks and people involved) do I have for this feature ?" Then you work wit…

Shape up is a powerful framework, it takes a while to get right but it’s very worth doing. We are about a year in, still improving at it. Our main focus now is doing things more by the book and getting rid of the customizations we added in the early days - so try and not make too many! :)

Email is in profile if you want to chat more on trade notes on it.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#47

Post by adamwathan » Wed, Oct 06, 2021, 9:55 AM UTC

alfonsodev wrote: post #31

Not to hijack the conversation, but I though some people interested in replacing estimations could find this useful. In order to replace estimation, we are trying Basecamp's Shapeup language and techniques [1] In a nutshell the thinking is reversed, instead of asking: - "how long would it take to implement X?" you ask: - "how much appetite (in weeks and people involved) do I have for this feature ?" Then you work wit…

We work this way as well and it is wonderful.

We've been doing Shape Up since July 2020 and have yet to fail to ship what we shape for each 6 week cycle, and while there's sometimes a bit of a crunch in the last few days before a release, everyone is generally working at a pretty relaxed pace.

My favorite side-effect of working this way as a leader is that by batching 6 weeks worth of planning effort, the team (and myself) can just be generally left alone to get into it without having to decide on the next priority every single week. Combined with async check-ins (that the team provide themselves every couple of days via Basecamp) instead of daily/weekly stand-ups, everyone generally has an empty calendar at all times.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#48

Post by gpmcadam » Wed, Oct 06, 2021, 9:55 AM UTC

Shameless plug but we wrote a free tool that you can pipe in Jira data and run monte-carlo simulations to help generate forecasts.

If you don't want to go through the process of logging in via Jira there is a demo where you can manually input the data needed.

See: https://agilytics.leanloop.co.uk

Re: How to replace estimations and guesses with a Monte Carlo simulation

#49

Post by rightbyte » Wed, Oct 06, 2021, 10:01 AM UTC

How is this different to just summing estimated value and variance?

I agree with the author that agile world zero effort estimates are pointless though.

"... engineers determine it by licking the tip of their finger and putting it up in the air.

Unfortunately, the only way to win at the estimation game is not to play it." "

Re: How to replace estimations and guesses with a Monte Carlo simulation

#50

Post by ImaCake » Wed, Oct 06, 2021, 10:07 AM UTC

A little bit of knowledge about stats and the appropiate language helps make this exercise easier. The R language is good at this kind of simulation. It comes with a whole bunch of random number generators out of the box. Here is how to estimate frequency of 7 from sums of two d6 dice rolls in R in two lines:

n table(round(runif(n,1,6))+round(runif(n,1,6)))[6]/n

To break this down:

`n We want two six sided dice so we sample a random uniform (continous) distribution and round to a whole number:

`round(runif(n,1,6))`

This will make a vector of 1 million integers representing random "rolls" of the dice. In R it is really easy to pair-wise add two vectors of the same length so to get two dice we do:

`round(runif(n,1,6))+round(runif(n,1,6))`

Now we have a million rolls of summed pairs of dice. We want the frequency count of 7s. We can get this by getting the frequency count of every dice with `table()`:

`table(round(runif(n,1,6))+round(runif(n,1,6)))`

Which will return a print out of frequencies for every number that appears in the vector.

We can select 7s by pulling the 6th index and then dividing by the `n` dice rolls to get the frequency:

`table(round(runif(n,1,6))+round(runif(n,1,6)))[6]/n`

We can do the exact same oneliner in python using numpy and the Counter() function from collections:

`Counter(np.around(np.random.uniform(1,6,size=10*6))+np.around(np.random.uniform(1,6,size=10*6)))[7]/10*6`

Will return the frequency of 7s in python3.