How to replace estimations and guesses with a Monte Carlo simulation • Hacker News Board

Re: How to replace estimations and guesses with a Monte Carlo simulation

#111

Post by toss1 » Wed, Oct 06, 2021, 3:39 PM UTC

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

>>I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date."

I hate to break it to you, but it happens a lot (and yes, management structures can often work to filter absolute bozos upwards). They make all kinds of representations to customers, investors, etc. about dates that have nothing approaching this level of estimation, more of a wishful-thinking goal. And when they fail, investments, customers, jobs, and even companies are lost — often unnecessarily if better plans and contingency plans had been made.

Sadly, just because it is blindingly obvious to you and me, does not mean it is so obvious to someone in a mgt chair.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#112

Post by mgkimsal » Wed, Oct 06, 2021, 3:41 PM UTC

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

> I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date."

Apparently I've worked with a lot of bozos over the last 25 years... (apologies to everyone I've worked with).

Re: How to replace estimations and guesses with a Monte Carlo simulation

#113

Post by zackmorris » Wed, Oct 06, 2021, 3:45 PM UTC

I wish we had public Monte Carlo simulations for the economy.

It's hard watching people argue about things with almost no evidence. The cynic in me is starting to think that that's all politics is. But imagine if people did have evidence, or at least projections within an error bound like the roughly 3% margin of error for polls, and could make decisions based on likely outcomes.

For example, how much would taxes on the US middle class decrease if taxes on the wealthy were raised? How much lower would they be after 5 years, 10 years, if that excess capital was reinvested in, say, an automation moonshot? I've framed a scenario which is obvious to me after running the simulations in my mind over 40 years of watching the national debt increase, but maybe I'm wrong (unlikely).

This site has a "Sample Size Calculator" tab which has a calculator for the number of samples needed to reach a certain confidence level:

https://www.pollfish.com/margin-of-error-calculator/

More info:

https://en.wikipedia.org/wiki/Margin_of_error

For a population size of 350 million, to get to a 95% confidence level with a 3% margin of error, requires polling 1068 people. Getting to 99% only requires 1849 people.

Now why can't we write something like the program from the article, maybe with some Bayesian inference, maybe some machine learning, with big sample sizes and high confidence levels? This may be the plot of some dystopian novel, but, surely more evidence would help inform a populace that's being spoon-fed ignorance by coopted media?

Re: How to replace estimations and guesses with a Monte Carlo simulation

#114

Post by dagw » Wed, Oct 06, 2021, 3:53 PM UTC

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date."

Happens all the time, and not just by bozos. Sometimes getting paid is entirely dependent on delivering when you said you would. I used to work at an animation studio and one year we worked on a series of Christmas themed TV spots. When we said we could deliver on time we where very much betting the farm our estimates where correct. Nobody is paying to air those in February.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#115

Post by canadianfella » Wed, Oct 06, 2021, 3:54 PM UTC

nanis wrote: post #101

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

While the article is rather low quality in and of itself, it still manages to point out: > Before claiming that the simulation above is reliable, I must expose a few things you’ll need to pay attention to when using a Monte Carlo Simulation to make forecasts: > (A) The quality of the inputs upon which your simulations depend. > (B) The consistency and predictability of your team. > (C) The need to re-forecast. > (D)…

> in and of itself

What’s the point of adding these words?

Re: How to replace estimations and guesses with a Monte Carlo simulation

#116

Post by benmanns » Wed, Oct 06, 2021, 4:07 PM UTC

zackmorris wrote: post #113

I wish we had public Monte Carlo simulations for the economy. It's hard watching people argue about things with almost no evidence. The cynic in me is starting to think that that's all politics is. But imagine if people did have evidence, or at least projections within an error bound like the roughly 3% margin of error for polls, and could make decisions based on likely outcomes. For example, how much would taxes on…

There are, roughly speaking. You can look at option pricing to see market predictions on various outcomes, at least ones that can be boiled down to "what will the price of SPY/TLT/EURUSD be in 2022?" If you have a better simulation, take your output and trade against the market where it's wrong.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#117

Post by luckydata » Wed, Oct 06, 2021, 4:14 PM UTC

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

this is an incredibly obvious yet valuable comment but all the replies seem to be working hard to miss the point.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#118

Post by groby_b » Wed, Oct 06, 2021, 4:18 PM UTC

xyzelement wrote: post #90

Many commenters on this story believe the main purpose of estimation is to assert when something will be done. Then they say "this is impossible" and call the whole exercise a waste of time, or management abuse. I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date." Benefits of estimation that are more…

> I gotta tell you that unless you're working with absolute bozos, nobody is looking at estimates and saying "oh duh, I am betting the farm this will complete on that date."

Meanwhile, anybody working anywhere close to the retail space fell laughing off their chair. Thanksgiving won't move no matter how much you insult your co-workers.

(This is not limited to retail. Many industries have fixed delivery dates and bet on those. Thanksgiving is just the most obvious example of an immovable object)

Re: How to replace estimations and guesses with a Monte Carlo simulation

#119

Post by jon-wood » Wed, Oct 06, 2021, 4:26 PM UTC

IshKebab wrote: post #89

jon-wood wrote: post #73

Earlier quoted context omitted.

Story points are a curse on the software development industry. However much people say "they're indicative, and don't map to hours", someone, somewhere, will map them to hours. The most accurate project plan I was ever involved in had only 3 values that could be assigned to a piece of work during the early estimation phase: hours, days, and weeks. Each of those was then turned into a range of possible hours they coul…

That seems reasonable, but I would say you don't have to resort to such crudeness. In my experience I know the difference between a task that will definitely take weeks and a task that might take a day or might take weeks. Just let me write that down! Even if you don't have any idea about the uncertainty we already have a crude way of measuring it - planning poker! Just record everyone's guesses instead of throwing a…

Sorry, reading back I definitely come across as disagreeing - I would kill someone for task tracking software which supports giving the degree of uncertainty on an estimate.

Re: How to replace estimations and guesses with a Monte Carlo simulation

#120

Post by jrek » Wed, Oct 06, 2021, 4:30 PM UTC

The issue I've always had with Monte Carl simulation in this context is that it takes more knowledge, expertise, time, and care to accurately perform a Monte Carlo Simulation than to prepare an accurate estimate. It's like saying you can avoid building (yet another) barely functional go-kart by instead building a four-wheel drive. Monte Carlo Simulation sometimes cloaks that the assumptions behind the simulation largely determine the output and are just as (if not more) prone to error as the usual assumptions.

If the burden of accurately estimating an average duration for a blog post is too much for your planner, what are the odds they're going to accurately develop a probability distribution for the duration of a blog post?

In simple estimates there's no need to use a stochastic approach at all. For instance, the example in the post - if you know a blog post takes between 1-10 days (uniformly distributed) and that you need to get one out every ~6 days to get 60 out in a year, you already know the probability is ~60%. If you know there's a skew to the higher end and guess a distribution (as in the example), again you can directly work out that the odds of success are ~35%.

There is value when the estimate is not corrupted by other priorities (rare), when the expertise to accurately develop distributions for activities exists (also rare, particularly for work that isn't easy to sample and re-forecast), and when the plan is complex enough that it's hard to directly predict the impact of your statistical assumptions.