Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

andersource.dev

Post reply on HN
Print view View original

181–190 of 308 posts

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#181

Post by mike_hearn »

russdill wrote: post #90
darawk wrote: post #86

Earlier quoted context omitted.

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand math…

I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. That is a problem and it does erode trust. But in some ways it's also how the sausage gets made, there's going to be room for things to get published they later gets refuted. People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becomi…

The reasonable middle ground is, or probably should be, closer to the latter end than the former end unfortunately.

Put simply, science is advertised as self-correcting but in reality it's not. Representative experience documented here: http://crystalprisonzone.blogspot.com/2021/01/i-tried-to-rep...

So, the reasons people learn a generalized distrust of science are that often the sausage doesn't get made. Bad science is published, applauded, cited, breathlessly covered in the media and may even be replicated, yet the first time outsiders to the field actually read the paper they realize it's nonsensical. But then they realize nobody cares because careers were made through this stuff, so why would anyone inside the field want to unmake them?

The degrading trust doesn't come from bad results per se, but rather the frequent lack of any followup combined with the lack of any institutional mechanisms to detect these problems in the first place beyond peer review, which is presented as a gold standard but is in no way adequate as such.

For example, consider how programmers use peer review. We use it, and we use lots of other tools too because peer review is hardly enough on its own to ensure quality. Now imagine you stumbled into a software company that held a cast-iron policy that because patches get reviewed by coworkers you simply don't need a test suite, nor manual testing, nor a bug tracker, code comments, security processes, strong typing, etc. And their promotion process is simply to make a ranking of developers by commit count and promote the top 10% every quarter, and fire the bottom 10%. Moreover they thought you were nuts for suggesting that there was any problem with this. You'd probably want to get out of there pretty fast, but, that's pretty much how (academic) science operates. So of course this degrades trust.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#182

Post by pastage »

lelanthran wrote: post #135
russdill wrote: post #90

Earlier quoted context omitted.

I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. That is a problem and it does erode trust. But in some ways it's also how the sausage gets made, there's going to be room for things to get published they later gets refuted. People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becomi…

> I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. It's more than that, I think. Sibling-thread poster hit the nail on the head when he complained of politicised science. The social sciences have this dominating and silencing effect on the rest of the sciences. There's always been junk science, and when found out it gets discredited. Th…

Group think exists in science too, I mean Newton calling Leibniz a copy cat was a set back and it's crazy that I still had to learn about the priority controversy almost 300 years later. We feed on unneccessary controversy.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#184

Post by roenxi »

russdill wrote: post #90
darawk wrote: post #86

Earlier quoted context omitted.

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand math…

I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. That is a problem and it does erode trust. But in some ways it's also how the sausage gets made, there's going to be room for things to get published they later gets refuted. People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becomi…

> Without becoming an expert yourself it's very difficult to know who or what to trust.

Who says there is anyone who can be trusted? People keep looking for leaders they can trust and it takes only a brief look at history to see that the search won't stop despite the jaw dropping futility of the exercise.

The important thing is to check that people have incentives to tell the truth and no conflicts of interest. I'd trust someone untrustworthy if they were making money off my well-being. The only thing to watch out for is them not being forthright about their incentives.

We shouldn't trust that skyscrapers stay up because engineers are trustworthy. They stay up because the engineer goes down with the building.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#185

Post by torginus »

When I saw the graphs in the original article I immediately came to a different conclusion - that people with a given amount of skill have low confidence in their ability to gauge how skilled they are compared to an arbitrary group.

For example, if someone gave me (or you) a leetcode-style test, and told me I'd be competing against a sample picked from the general population, and ask me how well I did, I'd probably rate myself near the top with high confidence.

Conversely, if my competitors were skilled competitive coders, I'd put myself near the bottom, again with high confidence.

Now, if I had to compete with a different group, say my college classmates, or fellow engineers from a different department, I'd be in trouble, if I scored high, what does that mean? Maybe others scored even higher. Or if I couldn't solve half of the problems, maybe others could solve even less - point is I don't know.

In that case the reasonable approach for me would be to assume I'm in the 50th percentile, then adjust it a bit based on my feelings - which is basically what happened in this scenario, and would produce the exact same graph if everyone behaved like that.

No need to tell tall tales of humble prodigies and boastful incompetents.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#186

Post by IshKebab »

jasonhansel wrote: post #38
parentheses wrote: post #30

What’s more interesting to me is what all the buzz over DK tells me. We are asymmetrically skeptical. In the same way as intelligent people doubt their own performance, they rightly doubt others’ performance. Maybe too much.

I think that most people who talk a lot about DK believe that they are the experts in one field or another. It serves mostly as a way of reassuring themselves of their own superiority. The message (for them) basically amounts to "other people's claim to knowledge is just further proof that they don't know anything."

Ha look at this guy saying things. What an idiot. DK effect amiright?

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#187

Post by whiddershins »

sally_glance wrote: post #131
hef19898 wrote: post #119

Earlier quoted context omitted.

>> figure skating, or law Thinking about all the "INAL" answers and the obviously wrong "legal" advice and opinion on legal topics you can come across on HN on a daily basis, I think law is good example of people overestimating their knowledge. Replace figure skating with any other sports, so, and try having discussions about, e.g, a defeat of any soccer team. All of a sudden everyone just became a soccer coach. And…

But is DK not explicitly about assessing yourself and not about assessing others? I feel like being able to critique the performance of others is different from being able to critique yourself. There might be a correlation between the two, but they're not the same.

Thinking you know enough to critique others is a self-evaluation.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#188

Post by mike_hearn »

danbruc wrote: post #179

Read the actual paper [1], there is so much more than those charts. They ask for an assessment of the own test score and an assessment of the ranking among the other participants to distinguish between misjudgments of the own abilities and the abilities of others. They give participants access to the tests of other participants and check how this affects self assessments - competent participants realize that they hav…

I think if you wanted to poke holes in the paper you'd start with the generic issues that are typical to much psychological research:

1. It uses a tiny sample size.

2. It assumes American psych undergrads are representative of the entire human race.

3. It uses stupid and incredibly subjective tests, then combines that with cherry picking:

"Thus, in Study 1 we presented participants with a series of jokes and asked them to rate the humor of each one. We then compared their ratings with those provided by a panel of experts, namely, professional comedians who make their living by recognizing what is funny and reporting it to their audiences. By comparing each participant's ratings with those of our expert panel, we could roughly assess participants' ability to spot humor ... we wanted to discover whether those who did poorly on our measure would recognize the low quality of their performance. Would they recognize it or would they be unaware?"

In other words, if you like the same humor as professors and their hand-picked "joke experts" then you will be assessed as "competent". If you don't, then you will be assessed as "incompetent".

Of course, we can already guess what happened next - their hand picked experts didn't agree on which of their hand picked jokes were funny. No problem. Rather than realize this is evidence their study design is maybe not reliable they just tossed the outliers:

"Although the ratings provided by the eight comedians were moderately reliable (a = .72), an analysis of interrater correlations found that one (and only one) comedian's ratings failed to correlate positively with the others (mean r = -.09). We thus excluded this comedian's ratings in our calculation of the humor value of each joke"

The fact that this actually made it into their study at all, that peer reviewers didn't immediately reject it, and that the Dunning-Krueger effect became famous, is a great example of why people don't or shouldn't take the social sciences seriously.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#189

Post by bmacho »

MichaelBurge wrote: post #56

The plot to me always read "People estimate themselves at 60-70% percentile - above average, but not the best". And then given this broad prior, people do place themselves accurately(because the plot is increasing). So it seems people are bad at doing global rankings. If I tried to rank myself amongst all programmers worldwide, that seems really hard and I could see myself picking some "safe" above-average value just…

> There's also: If you take 1 class in piano 30 years ago and can only play 1 simple song, that might put you in the 90th percentile worldwide just because most people can't play at all. But you might be at the 10th percentile amongst people who've taken at least 1 class. So doing a global ranking can be very difficult if you aren't exactly sure what the denominator set looks like.

Yes, and this literally implies that people in the lowest quartiles can't and won't rate themselves to be in the lowest quartiles when they are forced to give an answer. (Especially on tests that doesn't measure anything (getting jokes? really?), on tests that they have no knowledge about (how would they know that how their classmates perform on an IQ test???), or on tests that just have a high variance.)

And therefore they will "overestimate their performance".

It's like grouping a bunch of random people, and forcing them to answer whether their house is short, average or high. The "people living in short houses" will "overestimate the height of their houses", while the "people living in towers" will humbly say they live in an average high house.

Is this an existing and relevant psychological phenomenon, different from the general inability to guess unknown things? I don't think so.

If you think so, then give me proof.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#190

Post by mike_hearn »

kortilla wrote: post #142
whatshisface wrote: post #101

Earlier quoted context omitted.

> it's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods). I don't think this is true. It is possible to put a lot of work into unsound statistics and to make a lot of "noise and fury" about how mathematical you are while failing some basic principle, but I don't think sound statistics can mislead. The replication crisis was caused by scientists not being rigorous and journ…

> but I don't think sound statistics can mislead. Of course they can, unless you magically exclude all statistics that made a bad assumption on independence. I plot all the daily high temperatures and the presence of the ice cream cart and it turns out the ice cream cart causes warmer highs! Solid statistics. Turns out the guy that has the ice cream cart has a weather app on his phone though and doesn’t come out on f…

Is that the fault of statistics though, or the non-statistical implication of causation that was tacked on the end of the statistical detection of correlation? Statistics is pretty explicit that it can't tell you about causality, right?
Post reply on HN