Live data from Hacker News

Uniqueness Bias: Why it matters, how to curb it

arxiv.org

31–40 of 70 posts

Re: Uniqueness Bias: Why it matters, how to curb it

#31
I found the textbook example interesting, where the co-author with experience thought it would take 7+ years, and everyone else thought it would take about 2.

That makes a lot of sense to me, because I wrote a textbook in about 2 years. But...

There are textbooks and there are textbooks. My book covers exactly what I need for my class, has no publisher (students download it from the class website), and doesn't have a lot of the bells and whistles that a publisher would demand. I can easily see how a "real" textbook would take over 3x as long.

So part of the lesson of the authors' textbook is a standard software development lesson - there's a huge gap between a working prototype and a shipping product, and even if that prototype does everything you think the product needs to do, (a) you're wrong, and (b) you may be less than halfway to the point where you can ship.

Re: Uniqueness Bias: Why it matters, how to curb it

#32
> Sanbonmatsu et al. (1991) studied consumer choice between multiple brands and products. Experiments demonstrated that when deciding what to buy, consumers search for unique features instead of common ones among alternatives before making their choice. Bolton (2003) further found that even when decision makers are forced to consider more alternatives, including counterfactuals, decisions are still biased towards the initial scenario, which is typically based on availability bias and the inside view. Again, this supports our interpretation that uniqueness bias is likely the outcome of information processing by decision makers rather than an expression of hindsight bias.

This seems like a far-fetched conclusion to me. Neither of these two studies seem to suggest that people do not significantly overestimate uniqueness of projects in hindsight.

The 2003 study seems completely unrelated. How does bias torwards the inital scenario imply a uniqueness bias? That would require uniqueness to be the initial scenario. If anything, I would claim the opposite.

The 1991 study may imply that people tend to focus on unique aspects. Claiming that this tendency in product choice carries over to project management is a strecht imo. And even if it did, that would still not be a strong argument for the conclusion.

Re: Uniqueness Bias: Why it matters, how to curb it

#33
post #11

> perceived uniqueness is indeed highly statistically significantly associated with underperformance Isn't it possible that perceived unique projects are unique and therefore are harder and underperform? Did they control for that?

They did. Under "Perceived v. Actual Uniqueness" it says:

> we conclude from the available data that [...] perceived and actual uniqueness are uncorrelated

(Though as other comments suggest, their method of arriving at this conclusion might be flawed.)

Re: Uniqueness Bias: Why it matters, how to curb it

#34
post #8
post #5

Earlier quoted context omitted.

This goes beyond project estimations. The Metaculus tournaments have questions like > Will YouTube be banned in Russia before October 1? These are the kinds of questions most people I meet would claim are impossible to forecast accurately on because of their supposed uniqueness, yet the Metaculus community does it again and again. I believe the problem is a lack of statistical literacy. Back in the 1500s shipping ins…

I know you're coming from a good place, but the 'most people I meet don't understand this' line about statistics is quite arrogant. Most people you meet are fully capable of understanding statistics; you should do a better job of explaining it when it comes up, or maybe you are the one who misunderstands. After all, most statisticians thought Marilyn vos Savant was wrong about the goats too...

This is not at all true, and I think it's an example of what statisticians have to fight against in order to explain anything. Most people have an almost religious belief that inferences drawn from statistics should be intuitive, when they are often often extremely counterintuitive.

> After all, most statisticians thought Marilyn vos Savant was wrong about the goats too...

This is the opposite of the argument that you're making. Here you're saying that probability is so confusing and counterintuitive that even the experts get it wrong.

Re: Uniqueness Bias: Why it matters, how to curb it

#35
post #3

I read this book called "How Big Things Get Done." I've seen my fair share of projects going haywire and I wanted to understand if we could do better. The book identifies uniqueness bias as an important reason for why most big projects overrun. (*) In short, this bias leads planners to view their projects as unique, thereby disregarding valuable lessons from previous similar projects. (*) The book compiles 16,000 big…

I think a lot of modern US public works fall prey to politicians who think the objective of the project is the spending of the money. That is - they push for spending on transit so they can talk about how much, in dollars, they got passed in transit funding. The actual outcomes for many of them are, at best, inconsequential. Further cynicism could be layered in if you consider some of the blocks of donors (infrastruc…

> Further cynicism could be layered in if you consider some of the blocks of donors (infrastructure contractors / RE devs / etc) and blocks of voters (union transit workers & construction workers) who are recipients again of the spending but not the outcomes.

It's probably best to think of an initial budget as a foothold, and that its ideal amount is low enough to be approved, but high enough to prevent the organization from changing direction after realizing that it's not going to be nearly enough i.e. to think in terms of "pot-commitment."

Re: Uniqueness Bias: Why it matters, how to curb it

#36
post #24
post #15

Earlier quoted context omitted.

> Back in the 1500s shipping insurance was priced under the assumption that if you just learned enough about the voyage you could tell for certain whether it would be successful or not. Do you have a source on that? That sounds very unlikely to me. People just have to look at a single storm to see that it sometimes destroys some ships and does not destroy others. It is very clearly has a luck component. How could hav…

Okay, that was an oversimplification. I have long wanted to write more on this but it never comes out right. What they did was determine the outcome with certainty barring the will of God or similar weasels. I.e. they did thorough investigations and then determined whether the shipment ought to be successful or not, and then the rest laid in the hands of deities or nature, in ways they assumed were incalculable. This…

I think I got most of this from Willful Ignorance (Weisberg, 2014) but it was a while since I read it (and this was before I made more detailed notes) so I might be misremembering.

There are what looks like fantastic books on the early days of probability and statistics aside from Weisberg (Ian Hacking is an author that comes to mind, and maybe Stephen Stigler?) but I have not yet taken the time to read them – I'll read more and write something up on this some day.

But statistical understanding is such a recent phenomenon that we have been able to document its rise fairly well[1] which is fascinating in and of itself.

[1]: And yet we don't know basic stuff like when the arithmetic mean became an acceptable way to aggregate data. Situations that obviously (to our eyes) call for an arithmetic mean in the 1630s were resolved some other way (often midrange, mode, or just a hand-picked observation that seemed somehow representative) and then suddenly in the 1680s we have this letter to a newspaper which uses the arithmetic mean as if it was the obvious thing to do and then its usage is relatively well documented after that. From what I understand we don't know what happened between 1630 and 1680 to cause this change in acceptance!

Re: Uniqueness Bias: Why it matters, how to curb it

#38
post #22
post #3

I read this book called "How Big Things Get Done." I've seen my fair share of projects going haywire and I wanted to understand if we could do better. The book identifies uniqueness bias as an important reason for why most big projects overrun. (*) In short, this bias leads planners to view their projects as unique, thereby disregarding valuable lessons from previous similar projects. (*) The book compiles 16,000 big…

> 99.5% of those projects overrun their timeline or budget. This does not shock me. In the corporate world, plans are not there to be adhered to, but only to give upper management a feeling of having tightened the rope for those pesky engineers who wanted to work at a lazy pace. Such feeling usually vanishes as soon as reality kicks in.

I dunno, upper management normally move onto greener pastures long before reality comes crashing down. That does not happen until two managers over. But that's okay the new plan will fix everything.

Re: Uniqueness Bias: Why it matters, how to curb it

#39
post #8
post #5

Earlier quoted context omitted.

This goes beyond project estimations. The Metaculus tournaments have questions like > Will YouTube be banned in Russia before October 1? These are the kinds of questions most people I meet would claim are impossible to forecast accurately on because of their supposed uniqueness, yet the Metaculus community does it again and again. I believe the problem is a lack of statistical literacy. Back in the 1500s shipping ins…

I know you're coming from a good place, but the 'most people I meet don't understand this' line about statistics is quite arrogant. Most people you meet are fully capable of understanding statistics; you should do a better job of explaining it when it comes up, or maybe you are the one who misunderstands. After all, most statisticians thought Marilyn vos Savant was wrong about the goats too...

It's definitely arrogant. But after much experience trying to explain these things to people, I'm more and more convinced it's not just "if you just put it the right way they will understand". Sure, they will nod politely and pretend to understand, and may even do a passable job of reciting it, but once its cast in a slightly different light they are just as confused.

Much as with reading and writing, I think it takes an active imagination and a long slog of unlearning to trust logic (the "ought to" thinking that shields one from reality) and coming to terms with the race not being to the swift, etc, and that these effects can be quantified.

It's not that some people are incapable of it. Much like literal literacy has reached rates of 99.9 % in parts of the world, I'm convinced statistical literacy can too. But when your teacher is not statistically literate (which I hypothesise they are not, generally speaking), they will not pass that on to you. They will not give you examples where the race is not to the swift. They will not point out when things seem to happen within regular variation and when they seem to be due to assignable causes. They will not observe the battle against entropy in seating choices in the classroom. They will not point out potential confounders in propensity associations. They will not treat student performance as a sample from a hypothetical population. They will not grade multiple-choice questions on KL divergence, although that would be far more useful. I could go on but I think you get the point.

Yet to be clear, I'm not talking about just applying statistical techniques and tools. I'm talking about being able to follow a basic argument that rests on things like "yes I know they are a fantastic founder but startups fail 90 % of the time and so will they" or "if the ordinary variation between bus arrivals is 5–15 minutes and we have waited 20 minutes then there is something specific that put the bus we are waiting for into a different population." These are not obvious things to many people.

This is not a personal failure – it is a lack of role models and teachers. I wouldn't have considered myself statistically literate until recently, and only thanks to accidentally reading the right books. I wouldn't even have known what I was missing were it not for that!

I suspect it will take a few generations to really get it going.

If someone would donate me large amounts of money I would love to actively research this subject, come up with reliable and valid scales to measure statistical literacy, and so on. But in the meantime I can only think in my spare time and sometimes write about it online.

Re: Uniqueness Bias: Why it matters, how to curb it

#40
post #3

I read this book called "How Big Things Get Done." I've seen my fair share of projects going haywire and I wanted to understand if we could do better. The book identifies uniqueness bias as an important reason for why most big projects overrun. (*) In short, this bias leads planners to view their projects as unique, thereby disregarding valuable lessons from previous similar projects. (*) The book compiles 16,000 big…

This book is by Bent Flyvbjerg, the lead author of the paper being discussed here. He’s done a lot of great scholarship on how megaprojects go haywire, and uniqueness bias is definitely a big piece of it.

You especially see this bias with North American public transit. Most of our transit is greatly deficient compared to much of the rest of the world, and vastly more expensive to build (even controlling for wages). But most NA transit leadership is almost aggressively incurious about learning from other countries, because we are very unique and special and exceptional, so their far better engineering and project management solutions just wouldn’t work here and aren’t even worth considering.

Post reply on HN