Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uniqueness Bias: Why it matters, how to curb it

arxiv.org

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Uniqueness Bias: Why it matters, how to curb it

#41

Post by nuancebydefault »

tomlue wrote: post #8
kqr wrote: post #5

Earlier quoted context omitted.

This goes beyond project estimations. The Metaculus tournaments have questions like > Will YouTube be banned in Russia before October 1? These are the kinds of questions most people I meet would claim are impossible to forecast accurately on because of their supposed uniqueness, yet the Metaculus community does it again and again. I believe the problem is a lack of statistical literacy. Back in the 1500s shipping ins…

I know you're coming from a good place, but the 'most people I meet don't understand this' line about statistics is quite arrogant. Most people you meet are fully capable of understanding statistics; you should do a better job of explaining it when it comes up, or maybe you are the one who misunderstands. After all, most statisticians thought Marilyn vos Savant was wrong about the goats too...

I'm not surprised by the statement 'most people i meet don't understand this'. More than 50 percent of people I meet are less educated. Its statisticly evident.

I might be over reaching but in fact what comes across as arrogance is just an example of statistical illiteracy.

Most (>> 50 percent of) people are very good at detecting patterns. People are very bad at averaging numbers of events, because the detected patterns stand out so much and are implicitly and unconsciously exaggerated.

An example. "in my city people drive like crazy" in fact means: this week i was a lot on the road and i saw 2 out of 500 cars that did not follow the rules and there was even one honking. It 'felt' like crazy traffic but in fact it was not.

Re: Uniqueness Bias: Why it matters, how to curb it

#42

Post by nitwit005 »

steveBK123 wrote: post #30
ozgune wrote: post #3

I read this book called "How Big Things Get Done." I've seen my fair share of projects going haywire and I wanted to understand if we could do better. The book identifies uniqueness bias as an important reason for why most big projects overrun. (*) In short, this bias leads planners to view their projects as unique, thereby disregarding valuable lessons from previous similar projects. (*) The book compiles 16,000 big…

I think a lot of modern US public works fall prey to politicians who think the objective of the project is the spending of the money. That is - they push for spending on transit so they can talk about how much, in dollars, they got passed in transit funding. The actual outcomes for many of them are, at best, inconsequential. Further cynicism could be layered in if you consider some of the blocks of donors (infrastruc…

Personal suspicion is that real leadership is dull and thankless (like so many things in life).

Announcing a big new transit project is exciting. Actually running the program well requires a lot of boring study, meetings, and management of details. Why bother, if the voters don't punish them for not doing it?

You can find endless internet posts by people complaining their manager doesn't want to do the scheduling of employees, which is the most basic part of their job. It's too tedious, so they try to avoid it.

Re: Uniqueness Bias: Why it matters, how to curb it

#43

Post by LegionMammal978 »

kqr wrote: post #17
card_zero wrote: post #12

Earlier quoted context omitted.

> the Metaculus community does it again and again. Do they? I can't see where on that site there is something like "history of predictions" or "track record". Edit: here. https://www.metaculus.com/questions/track-record/ But now I can't tell if it's good or bad.

The binary calibration diagram is what I would focus on. Of all the times Metaculus has said there's an x % of something happening, it has happened nearly exactly x % of the time. That is both useful and a little remarkable!

It also seems gameable: for every big question of societal importance that people care about for its own sake, have a thousand random little questions where the outcome is dead obvious and can be predicted trivially. Would you know anything talking about weighing questions to account for this?

Re: Uniqueness Bias: Why it matters, how to curb it

#44

Post by christianqchung »

LegionMammal978 wrote: post #43
kqr wrote: post #17

Earlier quoted context omitted.

The binary calibration diagram is what I would focus on. Of all the times Metaculus has said there's an x % of something happening, it has happened nearly exactly x % of the time. That is both useful and a little remarkable!

It also seems gameable: for every big question of societal importance that people care about for its own sake, have a thousand random little questions where the outcome is dead obvious and can be predicted trivially. Would you know anything talking about weighing questions to account for this?

Trivial questions wouldn't result in a good histogram where a probability of 30% actually results in something happening roughly 1 in 3 times. Trivial would mean questions where the community forecast is 1% or 99%. Those are not the vast majority of questions on the site. It would be very boring if the site was 70% questions where the answer is obviously yes or obviously no.

Additionally, many questions require that you give a distributional forecast, in effect giving you 25/50/75th percentile outcomes for questions such as "how much will Bitcoin be with at the end of 2024?"

Who would be gaming the system here anyway, the site? Individual users?

Re: Uniqueness Bias: Why it matters, how to curb it

#45

Post by tomlue »

pessimizer wrote: post #34
tomlue wrote: post #8

Earlier quoted context omitted.

I know you're coming from a good place, but the 'most people I meet don't understand this' line about statistics is quite arrogant. Most people you meet are fully capable of understanding statistics; you should do a better job of explaining it when it comes up, or maybe you are the one who misunderstands. After all, most statisticians thought Marilyn vos Savant was wrong about the goats too...

This is not at all true, and I think it's an example of what statisticians have to fight against in order to explain anything. Most people have an almost religious belief that inferences drawn from statistics should be intuitive , when they are often often extremely counterintuitive. > After all, most statisticians thought Marilyn vos Savant was wrong about the goats too... This is the opposite of the argument that y…

My point was that the experts were blinded by arrogance.

Even back then most ( almost all? ) statisticians were capable of understanding the monte hall problem. Yet they just assumed that a woman was wrong when she explained something that didn’t match their intuition. Instead of stopping to think, they let their arrogance take over and just assumed they were right.

Re: Uniqueness Bias: Why it matters, how to curb it

#46

Post by avionical »

LegionMammal978 wrote: post #43
kqr wrote: post #17

Earlier quoted context omitted.

The binary calibration diagram is what I would focus on. Of all the times Metaculus has said there's an x % of something happening, it has happened nearly exactly x % of the time. That is both useful and a little remarkable!

It also seems gameable: for every big question of societal importance that people care about for its own sake, have a thousand random little questions where the outcome is dead obvious and can be predicted trivially. Would you know anything talking about weighing questions to account for this?

There's calibration, but you can also just see contests where you pit the community aggregate against individual forecasters and see who wins. The Metaculus aggregate is really dominant in this contest of predicting outcomes in 2023, for example. See this: https://www.astralcodexten.com/p/who-predicted-2023

Re: Uniqueness Bias: Why it matters, how to curb it

#47

Post by bsder »

ozgune wrote: post #3

I read this book called "How Big Things Get Done." I've seen my fair share of projects going haywire and I wanted to understand if we could do better. The book identifies uniqueness bias as an important reason for why most big projects overrun. (*) In short, this bias leads planners to view their projects as unique, thereby disregarding valuable lessons from previous similar projects. (*) The book compiles 16,000 big…

> Others reasons for slipping include optimism bias, not relying on the right anchor, and strategic misrepresentation.

Optimisim and misrepresentation are WAY more important.

Most engineers I know of were a bit optimistic to other engineers. This leads to slippage because subtasks have a finite amount they can come in early but an almost infinite amount of time they can come in late.

In addition, most engineers are acutely aware of what they think the project would take vs. what number management was willing to hear to launch the project.

Combine both of these and your project will never come in even remotely close to the estimates.

Re: Uniqueness Bias: Why it matters, how to curb it

#48

Post by atoav »

bsder wrote: post #47
ozgune wrote: post #3

I read this book called "How Big Things Get Done." I've seen my fair share of projects going haywire and I wanted to understand if we could do better. The book identifies uniqueness bias as an important reason for why most big projects overrun. (*) In short, this bias leads planners to view their projects as unique, thereby disregarding valuable lessons from previous similar projects. (*) The book compiles 16,000 big…

> Others reasons for slipping include optimism bias, not relying on the right anchor, and strategic misrepresentation. Optimisim and misrepresentation are WAY more important. Most engineers I know of were a bit optimistic to other engineers. This leads to slippage because subtasks have a finite amount they can come in early but an almost infinite amount of time they can come in late. In addition, most engineers are a…

And sometimes the reality is that the realistic answer to a time estimate is "If we are lucky it takes me 30 minutes, if we are unlucky 30 days".

E.g. when it turns out to your surprise, that a part you had in your hardware design was replaced with a part that was 5 cents cheaper, but uses a undocumented protocol that someone has to re-implement, so a goal that was trivial in theory has suddenly involves hardcore reverse-engineering in a high pressure environment. A thing all engineers love.

The only time someone can give you reasonably accurate estimates is when they do something that down to the tiniest detail they have done before. The problem with that is, that in software things change constantly. A thing that was trivial to do with library X and Component Y of version 0.9 might be a total pain in the rear with Library Z and Component Y of version 1.0.

But yeah, unexperienced engineers are going to be optimistic that it is possible, because in theory it should be trivial.

Re: Uniqueness Bias: Why it matters, how to curb it

#49

Post by bux93 »

zerodensity wrote: post #38
pif wrote: post #22

Earlier quoted context omitted.

> 99.5% of those projects overrun their timeline or budget. This does not shock me. In the corporate world, plans are not there to be adhered to, but only to give upper management a feeling of having tightened the rope for those pesky engineers who wanted to work at a lazy pace. Such feeling usually vanishes as soon as reality kicks in.

I dunno, upper management normally move onto greener pastures long before reality comes crashing down. That does not happen until two managers over. But that's okay the new plan will fix everything.

This is why your project shouldn't be too small.

If a project is projected to be finished in 6 months, the current manager will still be there, and the success or failure will reflect on their record. It can only go wrong and reflect badly on them.

If a project will take 3 years, the manager can already collect their points for initiating a project with an incredible business case and innovative approach, leave after 18 months, and after a further six month, the new manager can say 'wow, my predecessor left a big mess, I'll clean it up/kill it'.

Re: Uniqueness Bias: Why it matters, how to curb it

#50

Post by dspillett »

atoav wrote: post #48
bsder wrote: post #47

Earlier quoted context omitted.

> Others reasons for slipping include optimism bias, not relying on the right anchor, and strategic misrepresentation. Optimisim and misrepresentation are WAY more important. Most engineers I know of were a bit optimistic to other engineers. This leads to slippage because subtasks have a finite amount they can come in early but an almost infinite amount of time they can come in late. In addition, most engineers are a…

And sometimes the reality is that the realistic answer to a time estimate is "If we are lucky it takes me 30 minutes, if we are unlucky 30 days". E.g. when it turns out to your surprise, that a part you had in your hardware design was replaced with a part that was 5 cents cheaper, but uses a undocumented protocol that someone has to re-implement, so a goal that was trivial in theory has suddenly involves hardcore rev…

> And sometimes the reality is that the realistic answer to a time estimate is "If we are lucky it takes me 30 minutes, if we are unlucky 30 days".

“I'll put that in Project as 60 minutes, then if you are luck you've got double time for contingency.” -- the external consultant acting as project manager.

Been there before…

Never give a best case estimate, or anything close to, even when quoting a range. Some will judge you, or worse make plans around you, based on that and little else, and it *ahem* isn't their fault if things overrun.

Post reply on HN