Live data from Hacker News

I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

andersource.dev

171–180 of 308 posts

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#171

Earlier quoted context omitted.

> People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becoming an expert yourself it's very difficult to know who or what to trust. A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendatio…

>A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendations than doing nothing (e.g. replace fat with sugar). It's not the field that is bogus in that case. People were quite literally bribed to push this. This could happen anywhere, anytime, i…

That was just an example, but we can go into more detail - at every point, nutrition studies are highly questionable. Sample sizes are usually minuscule (you can find important studies with 5-10 subjects, and even the largest studies rarely have more than a few hundred), they are often biased samples (only overweight people, only people with heart disease or diabetes etc.), they often don't account for likely confounding factors (using weight without accounting for muscle mass, no accounting for stress, time in the sun etc.) and on and on. And all of this in a subject where we basically don't have any clear idea about how much metabolism differs from one person to the next, based on what factors (e.g. gut microbiome has only been recently identified as a major component of digestion that can differ significantly between people; psychological effects of diet are even less well understood, even though your food choices are obviously not coming from some pure realm of reason).

Basically the digestive system is far too complex for us to understand from first principles at this time. Your diet has a very complex and very slow effect on your body, with some exceptions. Numerous diseases and environmental factors impact how this plays out exactly. So, to do real research in nutrition, the only chance right now would be to conduct massive studies over long periods of time with rigorous controls on subjects' nutrition and activities - which is basically impossible, or at least prohibitively expensive.

Instead, we get conclusions drawn from studies of a few dozen people over a few months or at best a year (in "long-term" studies). Or, we get conclusions drawn from comparing diet across huge populations ("the Mediterranean diet", "the Japanese diet", "the American diet" etc) with no possibility to control for obvious differences in nutrients, environmental factors, lifestyle differences, access to healthcare etc. Both of these are worthless conclusions, they don't tell you anything at all.

The only successes of nutrition science have been identifying the most basic nutrients we need to survive at a basic level (protein, fat, carbohydrates, and the various vitamins and minerals). Basically nothing beyond that should be trusted.

As a fun historical note, after the discovery of the macro-nutrients there was a budding field of nutrition scientists confidently recommending optimal diets using scientific methods. Unfortunately, they had no idea about the existence of micro-nutrients, so actually following some of their diets you could actually end up getting scurvy or other serious malnutrition diseases. The current slew is not that bad, but I wouldn't be surprised if in the future we will look back similarly at some common diet advice of today.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#172
> We don’t need statistics to learn about the world.

A sentence, written by the author on, commented by me on and read by the HN community on devices, which exist only thanks to 80-90 years of rigorous, statistics based QA in engineering, especially in mechanical/hardware engineering.

Anyhow, after spending years on a team filled with social science PHDs, I would not waste my time on reading papers about statistical analysis done by social scientist.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#173

Earlier quoted context omitted.

I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. That is a problem and it does erode trust. But in some ways it's also how the sausage gets made, there's going to be room for things to get published they later gets refuted. People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becomi…

> I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. It's more than that, I think. Sibling-thread poster hit the nail on the head when he complained of politicised science. The social sciences have this dominating and silencing effect on the rest of the sciences. There's always been junk science, and when found out it gets discredited. Th…

No this is not new. You have always have a direction set by political views, even if we have decided they are wrong they are still hard to kill like: smoking is good, white people are superior. There is still "science" being done to bolster those political views.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#174

Beyond the validity of the statistical methods used.. can someone clarify what is the actual hypothesis we are debating about competence? And what does each article propose? My understanding is that the hypothesis is "Those who are incompetent overestimate themselves, and experts underestimate themselves". DK says: True DK is Autocorrelation says: ??? "I cant let go..." says: True? HN says: also True? Is there really…

> My understanding is that the hypothesis is "Those who are incompetent overestimate themselves, and experts underestimate themselves". The DK hypothesis is "double burden of the incompetent": "Because incompetent people are incompetent, they fail to comprehend their incompetence and therefore overestimate their abilities more than expertes underestimate theirs" Arguably the hypothesis that matches the data from the…

>Arguably the hypothesis that matches the data from the DK paper best is: "Everyone thinks they're average regardless of skill level"

No, if you look at the graph[0] everyone thinks they are above average (over 50). The worst think they are a little above average and everyone else thinks they are better and better but increasing by less than the real difference.

At any rate, the issue seems to be with how people imagine everyone performs - they seem to think there are a lot of people who are really bad for a start, and seemingly a bit more people who are really good than there are (at least if we assume the results are accurate).

0. https://andersource.dev/assets/dk-autocorrelation/dk.webp

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#175

I think the main point of this post is correct -- just because you can find the effect in random noise, doesn't mean it's not real phenomenon that happens in real life. But it's missing a nuance there: if an effect can be replicated with random noise, then it's not a psychological effect (e.g. something that you would explain as a human bias), but a statistical effect. E.g. regression towards the mean is a real effec…

To riff on one of the author's previous comments, if height was uncorrelated with age for 0-20 year olds, that would be very surprising, and hopefully we wouldn't need to make posts saying "the fact 20 year olds are just as likely to be 1 ft tall as 1 year olds is not a physical phenomenon, it's a statistical effect."

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#176
post #136

Earlier quoted context omitted.

If by “destroyed” you mean “billionaires will happily fund their contrarian research for years regardless of its peer reviews”…

No, I mean researchers and professors suddenly not getting any research grants anymore, suddenly getting fired from their tenured jobs, not being invited anymore to conferences, etc. What billionaire funds “contrarian” research?

There are some funding sources for biologists trying to prove biblical creation.

Think also of all the money funding "climate change isn't real" or "climate change is not anthropomorphic" which can be traced back to big oil money.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#177

> We don’t need statistics to learn about the world. A sentence, written by the author on, commented by me on and read by the HN community on devices, which exist only thanks to 80-90 years of rigorous, statistics based QA in engineering, especially in mechanical/hardware engineering. Anyhow, after spending years on a team filled with social science PHDs, I would not waste my time on reading papers about statistical…

I don't think your interpreting the sentence correctly, I think it's more correct to read it as:

> We don't [only] need [the formal discipline of mathematics known as] statistics to learn about the world.

Sure, there are things you can only functionally ascertain through statistical analysis. But not everything in the world needs rigorous statistics.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#178
post #86
post #62

The conclusion in the article: > Why so angry? I know I’ve taken this far too personally. I have no illusions that everything I read online should be correct, or about people’s susceptibility to a strong rhetoric cleverly bashing conventional science, even in great communities such as HN. But frankly, for the last few years, the world seems to be accelerating the rate at which it’s going crazy, and it feels to me a l…

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand math…

>The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs.

That's a very simplistic take on it. Bad science is a necessary part of the process and dealt with accordingly by the scientific method.

The problem is that science that is bad or incomplete is being reported as fact or truth, or arguably even worse, as entertainment in order to gain an audience. This is what actually eroded the trust in science, as people kept repeating things that reinforced and further misshaped their biases.

Human nature tends to distrust stuff we don't understand. Hence, our trust in science, which in many fields is often beyond the understanding of laymen, have to have constant reinforcement. However, the goal of media, and especially social media is to increase eyeballs for their content, and the truth sells a lot less well than sensationalised content pandering to the audience.

Simply put, there is not much profit in reporting science truthfully, and every incentive to sensationalise it.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#179
Read the actual paper [1], there is so much more than those charts. They ask for an assessment of the own test score and an assessment of the ranking among the other participants to distinguish between misjudgments of the own abilities and the abilities of others. They give participants access to the tests of other participants and check how this affects self assessments - competent participants realize that they have overestimated the performance of other participants and now assess their own performance as better than before, incompetent participants do not learn from this and also assess their performance even better than before. They randomly split participants into two groups after a test, give one group additional training on the test task, and then ask all of them to reconsider their self assessments - incompetent participants that received additional training are now more competent and their self assessment becomes more accurate. This is not everything from the paper and probably also somewhat oversimplified, I just want to provide a better idea of what is actually in there.

Everyone is free to question the results, but after actually reading the entire paper I can confidently say that poking a bit at the correlation in the charts falls way short of undermining the actual findings from the paper. The actual results are much more detailed and nuanced than two straight lines at an angle.

[1] https://www.researchgate.net/publication/12688660_Unskilled_...

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#180

tldr; D+K's experiment was: Assign the numbers 1 thru 10 to ten people. Have each role a 10 sided die. The person assigned a 1 will roll higher than his assigned number 90% of the time. Daniel: >It’s not a “statistical artifact” - that will be your everyday experience living in such a world. You can experience statistical effects. I think a lot of controversy comes from how Dunning and Kruger's paper leads people to…

The claim that skill does not exist or that people are totally unable to recognize how good they are at anything is quite radical.

You are sort of smuggling in the assumption for example that Olympian medalist lifters, when asked how much they can deadlift, will have the same distribution of answers as people who never deadlift (but are aware that totally sedentary men can probably deadlift like 200lbs and totally sedentary women can probably deadlift like 150lbs). If this were true, it would be worth publishing a paper about it.

It's sort of surprising to me to read your comment because TFA is an extended rebuttal of your comment.

> I think a lot of controversy comes from how Dunning and Kruger's paper leads people to interpret the data as hubris on the part of low-performers, and the statistical analysis demolishes that interpretation. Not knowing how well you performed is not the same thing psychologically as "overestimating" your performance.

D-K actually found that low performers were less accurate at assessing their skill than high performers, and the article you refer to obviously did not find this effect in random data, so I'm not sure how it was demolished.

Post reply on HN