Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knowledge overconfidence is associated with anti-consensus views

science.org

Post reply on HN
Print view View original

111–120 of 132 posts

Re: Knowledge overconfidence is associated with anti-consensus views

#111

Post by somenameforme »

mlyle wrote: post #110
somenameforme wrote: post #107

Earlier quoted context omitted.

Ignoring your attempt at predictably headed ad hominem, you again cannot make your 'conclusion' from the study. They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally did everything trying to massage their numbers into the conclusion they wanted to make. Ironically, the numbers don't even support that if you do use the subscales. In all studies the…

> Ignoring your attempt at predictably headed ad hominem ???? ad hominem? Where? I'm saying it's very difficult to quiz about the actual controversy itself, because then we get very close to the point of disagreement and risk confounding. The research explicitly mentions this aspect of design. > They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally…

You're definitely correct there on my misreading of the tables. To further clarify, I also decided to see precisely what "points" meant rather than continuing to just skim. They chose to rate true/false answers on a -3 to +3 scale driven by the respondent's certainty. This means one question wrong, out of 34, was able to drive up to a 6 point difference at max certainty. And so, somewhat serendipitously, my point, with some 'modification' remains that opposition in no case was rarely able to explain even a single field-specific question missed, at least not at max certainty.

As for the replicability - again, this study arbitrarily removed people fully in line with the consensus. That makes it fairly safe to say that replication is a nonstarter. But I'd also add that another red flag for social science papers is when they collect a large number of variables that end up having no relevance to the published conclusion. Large numbers of variables is a key resource in p-hacking. And this paper was collecting all sorts of data that went completely unused and had nothing whatsoever to do with what they ultimately chose to publish.

Re: Knowledge overconfidence is associated with anti-consensus views

#112

Post by mlyle »

somenameforme wrote: post #111
mlyle wrote: post #110

Earlier quoted context omitted.

> Ignoring your attempt at predictably headed ad hominem ???? ad hominem? Where? I'm saying it's very difficult to quiz about the actual controversy itself, because then we get very close to the point of disagreement and risk confounding. The research explicitly mentions this aspect of design. > They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally…

You're definitely correct there on my misreading of the tables. To further clarify, I also decided to see precisely what "points" meant rather than continuing to just skim. They chose to rate true/false answers on a -3 to +3 scale driven by the respondent's certainty. This means one question wrong, out of 34, was able to drive up to a 6 point difference at max certainty. And so, somewhat serendipitously, my point, wi…

> The effect of opposition on the binarized full set of 34 objective knowledge items variable:

> Estimate Std. Error df t value Pr(>|t|)

> (Intercept) 25.90592 0.43454 13.11332 59.617 > opposition -0.664790.06842 2130.90896 -9.717 Looks like each point of opposition was about 2/3rds of one more incorrect true/false question. So the people who were most opposed scored >3 questions worse on a 34 question test, on average.

> again, this study arbitrarily removed people fully in line with the consensus.

Median filtering and trimming saturated measures is common in research like this-- hopefully designed into the original protocol. I do agree it would be nice to see their preregistration.

But, weird things happen at the tails-- truncation effects, etc.

Re: Knowledge overconfidence is associated with anti-consensus views

#113

Post by somenameforme »

mlyle wrote: post #112
somenameforme wrote: post #111

Earlier quoted context omitted.

You're definitely correct there on my misreading of the tables. To further clarify, I also decided to see precisely what "points" meant rather than continuing to just skim. They chose to rate true/false answers on a -3 to +3 scale driven by the respondent's certainty. This means one question wrong, out of 34, was able to drive up to a 6 point difference at max certainty. And so, somewhat serendipitously, my point, wi…

> The effect of opposition on the binarized full set of 34 objective knowledge items variable: > Estimate Std. Error df t value Pr(>|t|) > (Intercept) 25.90592 0.43454 13.11332 59.617 > opposition -0.664790.06842 2130.90896 -9.717 Looks like each point of opposition was about 2/3rds of one more incorrect true/false question. So the people who were most opposed scored >3 questions worse on a 34 question test, on avera…

Like mentioned, I was more interested in the impact on the subtopic specific results because of what we discussed. Giving a quiz on American history to judge your knowledge of French history is obviously not a reasonable idea, even if the skill-set overlap would probably give at least some weak correlation. In the field-specific binarized case (which removes the -3 to +3 noise), the impact is 0.09 points. So that translates to a fraction of a question difference in results.

And while I'm aware trimming extreme outliers, regardless of the side they end up on, I'm unaware of any study entirely removing a segment from their sample, from one side only, which is critical to your entire hypothesis, and doing so without any explanation, let alone justification, whatsoever. One thing I'd observe is that the observed effect is small enough that if this culled group had a knowledge specific score below the mean, then it's likely that their entire conclusion would be invalid.

One other issue we have not discussed is the methodology not only resulted in a very non-representative sample, but also was indirectly testing something else. The surveys were done on the internet, and all of the general knowledge questions (besides the covid ones, which were just...) have answers which can be looked up in a matter of seconds.

Re: Knowledge overconfidence is associated with anti-consensus views

#114

Post by mlyle »

somenameforme wrote: post #113
mlyle wrote: post #112

Earlier quoted context omitted.

> The effect of opposition on the binarized full set of 34 objective knowledge items variable: > Estimate Std. Error df t value Pr(>|t|) > (Intercept) 25.90592 0.43454 13.11332 59.617 > opposition -0.664790.06842 2130.90896 -9.717 Looks like each point of opposition was about 2/3rds of one more incorrect true/false question. So the people who were most opposed scored >3 questions worse on a 34 question test, on avera…

Like mentioned, I was more interested in the impact on the subtopic specific results because of what we discussed. Giving a quiz on American history to judge your knowledge of French history is obviously not a reasonable idea, even if the skill-set overlap would probably give at least some weak correlation. In the field-specific binarized case (which removes the -3 to +3 noise), the impact is 0.09 points. So that tra…

> I was more interested in the impact on the subtopic specific results because of what we discussed.

I think they're both interesting. Broad overconfidence in performance in basic science combined with low performance is an interesting characteristic of a population. The fact that this is represented in those with contrarian views is interesting.

> So that translates to a fraction of a question difference in results.

The subscale finding, just shows that this same phenomenon appears to also extend to the subjects they are contrarian about. And it's about the same magnitude of effect, because there's very few questions on the subscale.

On the big test, a participant does about 2% worse per unit of disagreement. On the subscale, they do about 1.8% worse per unit of disagreement. The effect appears to have the same magnitude and is statistically significant in both cases.

This is an invitation to do studies on specific fields of disagreement with better samples and more questions on the subscale. But this early research casts a wide net across many types of anti-consensus view. Study 5 is a small, possibly flawed step in this direction.

I'd be particularly interested in a 2 variable analysis that attempts to model how one's overall objective performance and level of disagreement models objective performance in the specific field where they disagreed.

> outliers, ...

This is all moving goalposts when we were originally discussing the question set. I've already agreed that sociology and psychology research via Turk is problematic and faces many confounds and selection problems.

Re: Knowledge overconfidence is associated with anti-consensus views

#115

Post by somenameforme »

mlyle wrote: post #114
somenameforme wrote: post #113

Earlier quoted context omitted.

Like mentioned, I was more interested in the impact on the subtopic specific results because of what we discussed. Giving a quiz on American history to judge your knowledge of French history is obviously not a reasonable idea, even if the skill-set overlap would probably give at least some weak correlation. In the field-specific binarized case (which removes the -3 to +3 noise), the impact is 0.09 points. So that tra…

> I was more interested in the impact on the subtopic specific results because of what we discussed. I think they're both interesting. Broad overconfidence in performance in basic science combined with low performance is an interesting characteristic of a population. The fact that this is represented in those with contrarian views is interesting. > So that translates to a fraction of a question difference in results.…

Something I recently considered was that many of the questions, the "good" questions, do ultimately test ideology more than knowledge. The obvious example would be something like "Humans share the majority of their genes with chimps." This is of course factually true true, but it seems much more likely to test ideology than scientific knowledge, in part because of how it is phrased. Imagine the question were framed as "Scientists claim that humans share the majority of their genes with chimps." Now you are no longer testing agreement with the scientific view, but knowledge of it.

And perhaps a bigger issue is that there were literally zero questions that would do the same "trick" vice-versa for somebody who ideologically agrees with one of the topics, but otherwise lacked much knowledge. Testing this would have been quite easy by simply throwing in questions that sound like they fit the consensus but have an important aspect that makes them false. An example would be "Evolution is a process of improving a species which, over time, may lead one species to become an entirely new one." That is, of course, false - but somebody of low knowledge and high consensus agreement, would likely consider it to be true.

My personal hypothesis would be that extreme views tend to be associated with high confidence and relatively low knowledge. And this could be extremes of agreement or disagreement with a topic. As Betrand Russell put it, "The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt."

Re: Knowledge overconfidence is associated with anti-consensus views

#116

Post by mlyle »

somenameforme wrote: post #115
mlyle wrote: post #114

Earlier quoted context omitted.

> I was more interested in the impact on the subtopic specific results because of what we discussed. I think they're both interesting. Broad overconfidence in performance in basic science combined with low performance is an interesting characteristic of a population. The fact that this is represented in those with contrarian views is interesting. > So that translates to a fraction of a question difference in results.…

Something I recently considered was that many of the questions, the "good" questions, do ultimately test ideology more than knowledge. The obvious example would be something like "Humans share the majority of their genes with chimps." This is of course factually true true, but it seems much more likely to test ideology than scientific knowledge, in part because of how it is phrased. Imagine the question were framed a…

> This is of course factually true true

If it's factually true, it's not testing ideology.

> but it seems much more likely to test ideology than scientific knowledge, in part because of how it is phrased

Is there anyone credible who disagrees with the statement here?

I think a bigger issue is that it may be testing not knowledge but careful, critical reading. I can score well on this test, but my flippant quick answers are not very accurate. I am not surprised those that are worse at critical reading might be more likely to have anti-consensus views.

> My personal hypothesis would be that extreme views tend to be associated with high confidence and relatively low knowledge.

That doesn't seem to be the case here; while we don't know the "1" full agreement results, the rest seems to be a pretty dang linear trend line.

Re: Knowledge overconfidence is associated with anti-consensus views

#117

Post by somenameforme »

mlyle wrote: post #116
somenameforme wrote: post #115

Earlier quoted context omitted.

Something I recently considered was that many of the questions, the "good" questions, do ultimately test ideology more than knowledge. The obvious example would be something like "Humans share the majority of their genes with chimps." This is of course factually true true, but it seems much more likely to test ideology than scientific knowledge, in part because of how it is phrased. Imagine the question were framed a…

> This is of course factually true true If it's factually true, it's not testing ideology. > but it seems much more likely to test ideology than scientific knowledge, in part because of how it is phrased Is there anyone credible who disagrees with the statement here? I think a bigger issue is that it may be testing not knowledge but careful, critical reading. I can score well on this test, but my flippant quick answe…

Nobody is forced to accept a fact if they do not want to. There's a large gulf between awareness of knowledge, including facts, and agreement with such. This is an especially important nuance in this quiz, where the precise implications/meanings of knowledge vs ideology are a critical part of the study.

Imagine that there was a secular Islamic expert. And he was quizzed on Islamic ideology in a similar fashion, such that he was obligated to imply belief in such, in his answers. Assuming he is being as accurate as he can in his responses, he would end up scoring as completely ignorant of Islam as he marked all beliefs as false.

I disagree there's a clear trend here, even without the 1s, because there were clear exceptions such as e.g. climate change where, even by their metrics, anti-consensus views were predictive of greater knowledge. So you end up having to, at a minimum, limit the stated claim to specific fields.

Re: Knowledge overconfidence is associated with anti-consensus views

#118

Post by mlyle »

somenameforme wrote: post #117
mlyle wrote: post #116

Earlier quoted context omitted.

> This is of course factually true true If it's factually true, it's not testing ideology. > but it seems much more likely to test ideology than scientific knowledge, in part because of how it is phrased Is there anyone credible who disagrees with the statement here? I think a bigger issue is that it may be testing not knowledge but careful, critical reading. I can score well on this test, but my flippant quick answe…

Nobody is forced to accept a fact if they do not want to. There's a large gulf between awareness of knowledge, including facts, and agreement with such. This is an especially important nuance in this quiz, where the precise implications/meanings of knowledge vs ideology are a critical part of the study. Imagine that there was a secular Islamic expert. And he was quizzed on Islamic ideology in a similar fashion, such…

> Nobody is forced to accept a fact if they do not want to. There's a large gulf between awareness of knowledge, including facts, and agreement with such. This is an especially important nuance in this quiz, where the precise implications/meanings of knowledge vs ideology are a critical part of the study.

OK, well, here they expressed disagreement with stuff that is not just the scientific consensus, but the overwhelming agreement. If you want to phrase it as disagreeing rather than knowledge being lacking-- I think this is unhelpful. I do not think the nuance you're driving at here is worth complicating the questions or instructions.

> because there were clear exceptions such as e.g. climate change

No.

All 7 areas of controversy did worse on the entire test. It was not statistically significant for climate change and evolution. It was highly statistically significant for 4 of the remaining 5.

6 of the 7 did worse on their on subscales (significantly significant for 4 of these 6), with effect sizes of -.83, -.65, -.28, -.82, -.88, -.55; climate change had a non-statistically significant finding of an effect with slope .03. I do not consider this a counterexample.

You've been really advocating for study power and significance of results... when it favors your argument.

Re: Knowledge overconfidence is associated with anti-consensus views

#119

Post by somenameforme »

mlyle wrote: post #118
somenameforme wrote: post #117

Earlier quoted context omitted.

Nobody is forced to accept a fact if they do not want to. There's a large gulf between awareness of knowledge, including facts, and agreement with such. This is an especially important nuance in this quiz, where the precise implications/meanings of knowledge vs ideology are a critical part of the study. Imagine that there was a secular Islamic expert. And he was quizzed on Islamic ideology in a similar fashion, such…

> Nobody is forced to accept a fact if they do not want to. There's a large gulf between awareness of knowledge, including facts, and agreement with such. This is an especially important nuance in this quiz, where the precise implications/meanings of knowledge vs ideology are a critical part of the study. OK, well, here they expressed disagreement with stuff that is not just the scientific consensus, but the overwhel…

Argumentum ad populum my friend.

It is absolutely worth making the questions more clear, because this is the entire point of the study. Do people who do not agree with the consensus on something lack knowledge, or do simply not agree with said knowledge? Conflating the two in your own study does nothing but set yourself up to observe a tautology (those who disagree with a consensus, disagree with a consensus), which a cynic may interpret as not entirely accidental.

And no, I'm not referencing statistical power. If I haven't made myself clear, I think this study and their numbers, are both going to fall well into the endless black hole that is the replication crisis, which is especially pronounced in social psychology - which has an aggregate replication success rate in the 20s. What I am saying is that you can't make your broad claim based even what I suspect are deeply massaged figures.

Re: Knowledge overconfidence is associated with anti-consensus views

#120

Post by mlyle »

somenameforme wrote: post #119
mlyle wrote: post #118

Earlier quoted context omitted.

> Nobody is forced to accept a fact if they do not want to. There's a large gulf between awareness of knowledge, including facts, and agreement with such. This is an especially important nuance in this quiz, where the precise implications/meanings of knowledge vs ideology are a critical part of the study. OK, well, here they expressed disagreement with stuff that is not just the scientific consensus, but the overwhel…

Argumentum ad populum my friend. It is absolutely worth making the questions more clear, because this is the entire point of the study. Do people who do not agree with the consensus on something lack knowledge, or do simply not agree with said knowledge? Conflating the two in your own study does nothing but set yourself up to observe a tautology (those who disagree with a consensus, disagree with a consensus), which…

> Argumentum ad populum my friend.

Sorry, no. Overwhelming agreement along with readily self-observable characteristics is good enough. You can claim that the kidney is actually the largest organ of the human body, but everyone else agreeing is good enough-- especially when we can look at pictures from other people we trust, arrange to check a cadaver ourselves if we really care, etc.

I mean, I guess it's possible Big Skin (tm) has rigged all the measures and faked everything. /s

> What I am saying is that you can't make your broad claim based even what I suspect are deeply massaged figures.

One submeasure on one subpopulation having an outcome that does not support the claim but is also not inconsistent with that claim doesn't invalidate the claim.

You are talking about:

* A non-statistically significant finding

* Showing an opposite slope, but of tiny magnitude compared to the slopes in the other direction

* On one subpopulation

* On one submeasure.

Post reply on HN