Live data from Hacker News

I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

andersource.dev

291–300 of 308 posts

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#291

Earlier quoted context omitted.

> I don't recall a PC-mob being used to silence any and all non-supportive voices until quite recently. Are you genuinely serious or were you completely unaware of anti-communist government sanctioned blacklisting of academics suspected of being communist for political clout? Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Are you unaware of…

>Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Your point is good but this is a pet peeve of mine. Galileo was not punished by the church for saying the earth orbits the sun. Galileo was indicted and punished by the church because he was a local elite with several personal and political enemies within the church, and more directly, because…

"Galileo was not punished by the church for saying the earth orbits the sun. Galileo was indicted and punished by the church because he was a local elite with several personal and political enemies within the church, and more directly, because he slighted the pope, his former friend, by taking a philosophical argument made by said pope, and putting it, paraphrased, into the mouth of a character in his book who was named "simplicio" and cast as a moron."

This is not really true. While there is some truth to the claim that Galileo placed an argument made by Urban VIII in the mouth of Simplicio and that Urban took offense at this, the trial documents, especially Galileo's sentence, make it very clear that Galileo was being punished for heresy and the heresy he was being punished for was the notion that Sun did not move and that the Earth did. From the sentence:

> We say, pronounce, sentence, and declare that you, the abovementioned Galileo, because of the things deduced in the trial and confessed by you as above, have rendered yourself according to this Holy Office vehemently suspected of heresy, namely of having held and believed a doctrine which is false and contrary to the divine and Holy Scripture: that the sun is the center of the world and does not moved from east to west, and the earth moved and is not the center of the world, and that one may hold and defend has probable an opinion after it has been declared and defined contrary to Holy Scripture.

Note that the term "vehemently suspect" is technical term. The Roman Inquisition in the 17th century didn't generally deliver straight up and down guilty or not-guilty verdicts and rather organized convictions according to degrees of suspicion. "Vehement suspicion" indicated that there was at least some (but not much) degree of plausible deniability that Galileo didn't believe what he had written, and that was only because Galileo denied it to the court.

"That pope literally gave him permission to publish his claim that the earth orbited the sun, a claim which Galileo did not make based on science, but instead made because he felt the resulting (incorrect and based on outdated observations) mathematical model for the orbits of planets was more "elegant"."

No, Galileo was given permission to publish a book that presented a neutral comparison of the Copernican and Ptolemaic models on mathematical grounds with the intention of proving that the Church was justified in its suppression of Copernicanism. Galileo's book was not neutral--it argued heavily in favor of Copernicanism--and that's why he got in to trouble. Urban VIII had been his friend prior to this episode so it's likely that had Galileo not placed Urban's argument in Simplicio's mouth at the end that Urban would have protected him rather than punished him, but the reason that Simplicio was given that argument was because that argument was intended to be the end of the book. After four days of continuously losing the debate, Simplicio finally raises Urban's argument about the omnipotence of God and his opponents are forced to agree with him. The idea being that Galileo could stick to the letter of his remit, while still arguing what he wanted. Unfortunately he argued too well and readers realized where is real sympathies lay. Urban VIII was accused of protecting heretics (not just Galileo, but also but supporting the French against the Hapsburgs in the 30 years war,) and so he made an example of Galileo.

Galileo's arguments were not simply mathematical. In fact, if they were, he would never have been punished because it was already permissible to treat Copernicanism as a purely mathematical hypothesis; the 1616 prohibition of Copernicanism explicitly carved out that exception. But Galileo used a wide variety of arguments, including physical arguments. Galileo after all, was primarily what we would call a physicist rather than an astronomer. It was Galileo's insistence that Copernicanism must be physically true and not just a better mathematical model that made him a heretic in the eyes of the Sacred Congregation.

It's important to note that the quality of Galileo's scientific arguments were never a subject of his trial. It was only his conclusions that the Sacred Congregation took issue with. Galileo's chief (but not only) argument, from the tides is now considered to have been spectacularly wrong, but at no point did that come up in the trial. Galileo's arguments could have been 100% perfect and unassailable (and scientific arguments rarely are) and he would have still been punished.

"If the church truly wanted to punish him, they would not have sentenced him to literally stay at home in a beautiful villa and write books all day."

He was sentenced to life imprisonment. That was commuted to house arrest on account of his old age and not at his own house at first. He was prohibited from receiving medical attention late in life. All books by him were placed on the index and he was prohibited from taking visitors. He did manage to continue his work, but that was by publishing in the Netherlands which was a Protestant country and hence outside the reach of the Church. He had first tried to publish in Venice which was a hotbed of anti-clericalism and usually Inquisitorial orders, but even they would publish him.

"His official charge was that lay people are not allowed to interpret the scripture, which he did a bit in his book."

That was not the official charge. I quoted the official charge above. Galileo's interpretation of scripture happened much earlier and preceded the 1616 prohibition of Copernicanism. There Galileo gave counter argument as to why Copernicanism didn't contradict scripture and that served as the catalyst for the investigation that led to the prohibition. Galileo ultimately was not censured for writing on scripture and his argument was even well received to a degree (he had checked it with a cardinal before publishing it,) but the Sacred Congregation decided that it was more concerned about undermining the authority of the Church Fathers, many of whom took the famous passage from Joshua literally, than it was about accidentally hooking scripture to a provably false view of the world. If you read Cardinal Bellarmine's response to Foscarini regarding Galileo's letter, you'll see him very clearly cite the authority of the Church Fathers as his primary consideration. Bellarmine controlled the Sacred Congregation at the time so his opinion on the matter was the Church's opinion.

"The church did not care if you made mathematical or scientific arguments about how the world worked.T hey only cared that you leave theology to the priests. "

The decree from the Index of Forbidden Books banning Copernicanism:

> This Holy Congregagtion has also leaned about the spreading and acceptance many of the false Pythagorean doctrine, altogether contrary to Holy Scripture, that the moves and the sun is motionless, which is also taught by Nicolaus Copernicus' On the Revolution of the Heavenly Spheres and by Dieage de Zuniga's On Job. This may be seen from a certain letter published by a certain Carmelite Father, whose title is Letter of the Reverend Father Paolo Antonio Foscarini, on the Pythagorean and Copernican opinion of the Earth's Motion and the Sun's Rest and on the new Pythagorean World System in which the said Father tries to show that the abovementioned doctrine of the sun's rest at the center of the world and of the earth's motion is consonant with the truth and does not contradict Holy Scripture. Therefore, in order that this opinion may not advance any further to the prejudice of the Catholic truth, the Congregation had decided that the books by Nicolaus Copernicus and by Diego de Zuniga be suspended until corrected; but of the Carmelite Father Paolo Antonio Foscarini be completely prohibited and condemned; and that all other books which teach the same be likewise prohibited, according to whether the present decree it prohibits, condemns and suspends them respectively.

Note that Diego de Zuniga and Paolo Foscarini are both priests. This wasn't about keeping theology to the priests, it was about prohibiting certain theology that would undermine the authority of the Church. The correction applied to Copernicus's book is that it be changed to suggest that his system was not intended as a literal interpretation but only as a mathematical model. As I mentioned earlier, an allowance for treating with Copernicanism as a pure mathematical contrivance for the convenience of astronomers was made but treating it as literally true was declared "error", and later upgraded to "heresy" during Galileo's 1633 trial.

Sorry for the essay, but this subject is a pet-peeve of mine.

Some good books on the subject: 1. Behind the Scenes at Galileo's Trial - Richard J, Blackwell 2. The Essential Galileo - Maurice A. Finocchiaro 3. Galileo Heretic - Pietro Redondi

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#292
post #189

The plot to me always read "People estimate themselves at 60-70% percentile - above average, but not the best". And then given this broad prior, people do place themselves accurately(because the plot is increasing). So it seems people are bad at doing global rankings. If I tried to rank myself amongst all programmers worldwide, that seems really hard and I could see myself picking some "safe" above-average value just…

> There's also: If you take 1 class in piano 30 years ago and can only play 1 simple song, that might put you in the 90th percentile worldwide just because most people can't play at all. But you might be at the 10th percentile amongst people who've taken at least 1 class. So doing a global ranking can be very difficult if you aren't exactly sure what the denominator set looks like. Yes, and this literally implies tha…

One difference is that as the experiments were run on psychology students, they know the population, those are their peers with whom they interact on a daily level and they should have an idea of how they compare with them.

> how would they know that how their classmates perform on an IQ test???

Are you serious? If you're interacting with your classmates, you definitely should have some idea on how their intellectual capabilities differ between each other and also with respect to you. In a small class doing lots of things together, someone might even literally count their "ranking" at some metric that highly correlates with IQ, estimating that Bob, Jane and Mary are above me and Dan and Juliet are below me, so I'm at 40th percentile.

It's not appropriate to treat these aspects as unknown things or unknowable things.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#293

Earlier quoted context omitted.

The charts make it clear that people's self-assessment was (roughly) independent of their skill level. It's not obvious that students' self-assessment would be mostly random / unrelated to skill level. For me that's a non-obvious result. If people wander off through the verbiage of any article, where the chatter isn't supported by data, sure, they'll tend to get speculation.

I don't really understand what you're saying. Are you the saying the charts don't actually make it clear, or that they make it clear that self assessment is independent but not necessarily uniform?

The charts make it clear that self-assessment was roughly unrelated to ability. That's not an artifact of autocorrelation, instead it appears to be an experimental result.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#294
post #284

Earlier quoted context omitted.

> The tasks aren't arbitrary. They're meant to be a proxy for some universal concept of competence. I’ve seen absolutely nothing suggesting this. It’s explicitly about task competency; no particular task is specified nor needs to be specified. > That's why DK is a well known effect, it claims to hold true for anything even though they can't test every possible task. Yes, they claim it holds true for everything becaus…

The reason no particular task needs to be specified to invoke DK is exactly because they argue that their initial selection of experimental tasks is so general, that the effect must apply to everything. It feels like you and danbruc are inverting causality here. You start from the assumption that DK is a real effect and then say, because it's real and general, it doesn't matter what tasks they used to prove it. But t…

And because they claim it's both large and very general, they need to present evidence to support both these ideas.

To me the claims in the paper do not really seem that strong, almost to the point that I am not sure if they claim anything at all. If you read through the conclusions, they mostly report the findings of their experiments. The closest thing to any claims about generality I can find is that they discuss in which scenarios their findings will not apply. You could maybe read into this that they claim that in all other scenarios their findings apply, but that is not what they actually do.

But I guess the better way to discuss this is that you just quote the claims from the paper that you consider too strong and unjustified instead of me trying to anticipate what you are referring to or me going over each claim in the paper.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#295

Read the actual paper [1], there is so much more than those charts. They ask for an assessment of the own test score and an assessment of the ranking among the other participants to distinguish between misjudgments of the own abilities and the abilities of others. They give participants access to the tests of other participants and check how this affects self assessments - competent participants realize that they hav…

I think if you wanted to poke holes in the paper you'd start with the generic issues that are typical to much psychological research: 1. It uses a tiny sample size. 2. It assumes American psych undergrads are representative of the entire human race. 3. It uses stupid and incredibly subjective tests, then combines that with cherry picking: "Thus, in Study 1 we presented participants with a series of jokes and asked th…

I can't quite figure out from this post and the posts after if you have any background in social science or not (you have stated you didn't do social science professionally - but I get a nagging feeling you have studied it) - and I'll try to explain why I think it matters. For what it's worth - I wouldn't necessarily object to what you wrote here if you finished with "great example of why people don't take the social sciences seriously" and left it there. I do have a problem with "shouldn't", although in a different setting (i.e. amongst social science people) I would probably argue for "shouldn't".

Full disclaimer - I was a sociological researcher before I started working in IT - and would (I can appreciate the irony given all of this is about DK effect) rate myself as very significantly above average in terms of methodological rigour and mathematical skill compared to other social researchers.

One thing that is taught to social researchers - although I've seen it much less with psychologists - is that social research is fundamentally different from natural sciences in that it is accepted as fundamentally subjective. Now, a radical such as myself will tell you that all research, including natural science, is not entirely objective due to very subjective navigation of selection bias, but putting that to the side - this is an extremely important point when evaluating social research.

Coming back to your original point - I would agree with the points you object to vis-a-vis original DK Effect paper, however, as a social researcher, I am always already coming into reading that paper knowing that I'll have to take it with spoonfulls of salt. There is no need to write the paper in a way that puts in many of the disclaimers you might expect, because we are institutionally taught that these disclaimers apply.

Having said that - one of my peeves with social research, and why I ultimately went away, is that a lot of garbage goes on and gets through peer review. There is almost no proper testing of quantitative instruments and methods. Which is why I agree with your point that it rightfully isn't taken seriously - but I would object to your assertion that it shouldn't be taken seriously. Especially amongst IT professionals who are already going to have a bias against non-STEM. Point out the shortcomings and apply a different interpretive lense, rather than discounting the field completely - as social science can be better and taken seriously if it was held to a higher standard, even with the methodological shortcomings we have today - but it is very often discounted wholesale, which I don't think is going to incentivise the bubble that is forming around it to reform and get better.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#296

The plot to me always read "People estimate themselves at 60-70% percentile - above average, but not the best". And then given this broad prior, people do place themselves accurately(because the plot is increasing). So it seems people are bad at doing global rankings. If I tried to rank myself amongst all programmers worldwide, that seems really hard and I could see myself picking some "safe" above-average value just…

Thank you! This article creates a dichotomy where our hypothesis must be either 1) self-assessment is perfectly correlated with skill, or 2) completely uncorrelated. I think neither of these makes sense as a null hypothesis. The model you describe matches my intuition about what we should expect: people know something about their own skill level, but not everything.

One minor correction - the article creates a dichotomy where the hypothesis must be either 1) self-assesment is somewhat correlated with skill, or 2) completely uncorrelated

And this is a true dichotomy. The "autocorrelative" effect doesn't need perfect correlation, just some correlation.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#297
post #291

Earlier quoted context omitted.

>Are you unaware of churches excommunicating Galileo for daring to scholarly research into the earth rotating around the sun? Your point is good but this is a pet peeve of mine. Galileo was not punished by the church for saying the earth orbits the sun. Galileo was indicted and punished by the church because he was a local elite with several personal and political enemies within the church, and more directly, because…

"Galileo was not punished by the church for saying the earth orbits the sun. Galileo was indicted and punished by the church because he was a local elite with several personal and political enemies within the church, and more directly, because he slighted the pope, his former friend, by taking a philosophical argument made by said pope, and putting it, paraphrased, into the mouth of a character in his book who was na…

My "interpretation" of the situation comes mostly from http://tofspot.blogspot.com/2013/08/the-great-ptolemaic-smac...

Have you read this perspective before? Is it a well known "oh not this again" incorrect source in the world of renaissance italy theology and history?

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#298
post #89

Earlier quoted context omitted.

Completely agreed. While I strongly believe in citizen science and people's right (and perhaps obligation) to critique established science, there is just so much poor analyses done by people to criticize some scientific findings they do not approve, motivated emotionally or otherwise. This phenomenon does not bring us closer to finding scientific facts or resolving the replication crisis. People should learn some bas…

We should really just do (and certainly publish) less research.

I can see why this has been downvoted, but I didn’t mean to sound like a luddite. I really just think there is a lot of low quality research, especially in the social sciences, done primarily to keep an publishing schedule up for pressured academics. We’d often be better to spend more time thinking about and planning fewer better studies.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#299

Earlier quoted context omitted.

Even people conditioned to be humble could have a strong motivation to believe something is true and overestimate their own knowledge/ability in order to stand on what they perceive as evidence. For example a person's depression, religious beliefs, or an over-emphasized belief in DK itself could be a possible reason they have an erroneously deflated opinion of themselves, and simultaneously employ inflated confidence…

> I un-learned things like "all or nothing thinking" and the expectation that I could accurately predict the outcome of any course of action Do you know any words, stereotypes, or clichés for this? Or even what the related mental disorder is called if it were to become debilitating? Or a specific word for the complete clustering of related signals? I am guessing those issues plus there related issues (¿syndromic?) ar…

Individual layer:

It is closely associated with having a highly systematizing mind. People with ASD get drawn and pushed down a particular life history corridor. There are rewards of parental/teacher approval for high-performance in an area of profound interest, and a punishment in thef orm of peer bullying for low social skills. This conditions them to operate this way to avoid bullying and optimize for time seemingly well-spent with these impersonal systems. To justify one's own existence, there is this urge to live in a world where a narrowly focused mind is able to predictably produce an ideal world through expertise (all), and a tendency towards refusal to live outside of that, sometimes advancing into self destructive behaviour should that not be an available outcome (nothing). Getting ALL is not only about having things one wants, it is also about seeing the system work, and about identity, a sense of vindication.

Collective layer:

We see so much of it I think even among neurotypicals because we live in an extremely systematized world. Every single aspect of our world is seen by the "haves" of our society as a candidate for profitable separation through a digital layer. Anyone can end up metastasizing a systematizing mind. They just need to exhibit a hyper-focus on something impersonal and complex. As a global civilization, we have been doing this to ourselves and strapping others into it as much as we can. Mostly, only those living in rural parts of materially impoverished nations are spared this temptation.

Mental Symbolism layer:

It is symbolically speaking one entrance into a realm of mental death. With the devotion to lifeless systems the human is de-personalized, atomized, depressed, unrelational. They leave unfulfilled the inescapable truth of what it means to be human. To live your entire life this way is to betray your parents, ancestors, any friends or lovers you ever had, and anyone you could have helped.

Spiritual layer:

The pattern is quite literally satanic. The person has chosen to reign in this hell (both all and nothing), rather than serve in heaven (the humble, narrow middle path). The most powerful and beautiful of all created entities, with an astonishingly powerful mind, insisting on dwelling in a state of supreme perfection betrays the Father to become an engine of extinction.

Re: I can't let go of “The Dunning-Kruger Effect is Autocorrelation”

#300

Earlier quoted context omitted.

> I un-learned things like "all or nothing thinking" and the expectation that I could accurately predict the outcome of any course of action Do you know any words, stereotypes, or clichés for this? Or even what the related mental disorder is called if it were to become debilitating? Or a specific word for the complete clustering of related signals? I am guessing those issues plus there related issues (¿syndromic?) ar…

Individual layer: It is closely associated with having a highly systematizing mind. People with ASD get drawn and pushed down a particular life history corridor. There are rewards of parental/teacher approval for high-performance in an area of profound interest, and a punishment in thef orm of peer bullying for low social skills. This conditions them to operate this way to avoid bullying and optimize for time seeming…

Thank you. Your explanation is not what I expected, and I really appreciate it.
Post reply on HN