Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knowledge overconfidence is associated with anti-consensus views

science.org

Post reply on HN
Print view View original

121–130 of 132 posts

Re: Knowledge overconfidence is associated with anti-consensus views

#121

Post by somenameforme »

mlyle wrote: post #120
somenameforme wrote: post #119

Earlier quoted context omitted.

Argumentum ad populum my friend. It is absolutely worth making the questions more clear, because this is the entire point of the study. Do people who do not agree with the consensus on something lack knowledge, or do simply not agree with said knowledge? Conflating the two in your own study does nothing but set yourself up to observe a tautology (those who disagree with a consensus, disagree with a consensus), which…

> Argumentum ad populum my friend. Sorry, no. Overwhelming agreement along with readily self-observable characteristics is good enough. You can claim that the kidney is actually the largest organ of the human body, but everyone else agreeing is good enough-- especially when we can look at pictures from other people we trust, arrange to check a cadaver ourselves if we really care, etc. I mean, I guess it's possible Bi…

When Galileo made his case for a heliocentric universe, there was no silver bullet he offered. He simply personally felt it to be more probable than the geocentric model in spite the overwhelming evidence to the contrary given the evidence of the time. Would you thus claim he was lacked knowledge (of the geocentric universe) if he chose to respond false on a question "True/False Everything in the universe revolves around the Earth?" Measuring agreement is different than measuring knowledge.

As for your hypothesis, what would you propose would be a null hypothesis to test your hypothesis? It seems to me that it would be "Somebody who opposes a scientific consensus will score about the same in a test of general knowledge as somebody who supports it." And the climate change example confirms that null hypothesis, which would thus reject your hypothesis.

Re: Knowledge overconfidence is associated with anti-consensus views

#122

Post by mlyle »

somenameforme wrote: post #121
mlyle wrote: post #120

Earlier quoted context omitted.

> Argumentum ad populum my friend. Sorry, no. Overwhelming agreement along with readily self-observable characteristics is good enough. You can claim that the kidney is actually the largest organ of the human body, but everyone else agreeing is good enough-- especially when we can look at pictures from other people we trust, arrange to check a cadaver ourselves if we really care, etc. I mean, I guess it's possible Bi…

When Galileo made his case for a heliocentric universe, there was no silver bullet he offered. He simply personally felt it to be more probable than the geocentric model in spite the overwhelming evidence to the contrary given the evidence of the time. Would you thus claim he was lacked knowledge (of the geocentric universe) if he chose to respond false on a question "True/False Everything in the universe revolves ar…

I'm getting less and less interested in galloping around talking about different minutiae. I basically disagree with your entire argument. But:

> And the climate change example confirms that null hypothesis, which would thus reject your hypothesis.

???! fails to reject the null hypothesis. This seems to be a rather large and fundamental error. You don't ever "confirm[ the] null hypothesis"-- and you certainly don't in a way that lets you exclude other explanations.

e.g. I flip a coin twice. It is heads once and tails once. I perform a statistical test and this outcome is consistent with random chance. In no way does this let me reject the idea that the coin is unfair.

(Indeed, doing a trial of 4 flips and getting 4 heads in a row is consistent with random chance; such a trial would not be likely to increase my belief that the coin is fair, even if it is not strong statistical evidence that the coin is unfair).

Back to our case: we have a whole lot of strong results in one direction, and one ambiguous near-zero result in one submeasure in one subpopulation. Perhaps that subpopulation really is different. It's an interesting thing to measure again. But if you measure enough things, you can expect to find some of these in your result set purely by chance.

Re: Knowledge overconfidence is associated with anti-consensus views

#123

Post by somenameforme »

mlyle wrote: post #122
somenameforme wrote: post #121

Earlier quoted context omitted.

When Galileo made his case for a heliocentric universe, there was no silver bullet he offered. He simply personally felt it to be more probable than the geocentric model in spite the overwhelming evidence to the contrary given the evidence of the time. Would you thus claim he was lacked knowledge (of the geocentric universe) if he chose to respond false on a question "True/False Everything in the universe revolves ar…

I'm getting less and less interested in galloping around talking about different minutiae. I basically disagree with your entire argument. But: > And the climate change example confirms that null hypothesis, which would thus reject your hypothesis. ???! fails to reject the null hypothesis. This seems to be a rather large and fundamental error. You don't ever "confirm[ the] null hypothesis"-- and you certainly don't i…

With all due respect, that feels like a dodge. I decided to look at the result data for this survey [1]. I was unsurprised to find, in a simple skim of only the first survey, at least 2 comments mentioning this issue we're discussing. Asking an e.g. creationist "True/False The Earth is about 4.5 billion years old." and "True/False Scientists claim that the Earth is about 4.5 billion years old." are questions that are going to get very different answers. One measures assent with consensus, one measures knowledge of it. At least 2 people actually manually typing the issue out in a relatively small sample (for this sort of arbitrary specificity) means this is an extremely significant issue.

My argument, in the most specific form, is that I do not believe this study measures, or was intended to measure, what the authors' claim it does. This issue is quite subtle, but I'm inclined to believe in malice over incompetence when these authors have spent years working these numbers [2]. The current study is a rehash of a study they did two years ago. The max assent group does indeed demonstrate reduced knowledge / increased self perceived knowledge. And there are demographic biases abounds. The desired trend varies by an order of magnitude in countries outside the US.

So this study took the most extreme sample from a subpopulation they were already familiar with the results of, stripped out the subpopulation of that subpopulation which challenged their hypothesis, added in a number of questions that were more likely to result in noise than anything else (the population of those taking surveys for pennies a piece on Amazon Turk, that can explain the difference between e.g. UVA and UVB radiation is certainly near zero so true/false just gives you noise and randomness), and then published. Another fun sample bias: Literally every single person who took fewer than 3 minutes to complete the survey (again based on survey 1) and bothered to fill in some (generally random) answers indicated max dissent. An interesting issue in itself, but a further problem in this study given relatively few people indicated max dissent, so they're measuring people literally racing through the survey for their $0.85 likely without reading it beyond looking for the "Please select #2 as the answer to this question." questions.

[1] - https://osf.io/x23c8/?view_only=29a92a9a707547149f210e5bf76a...

[2] - https://osf.io/t82j3/

Re: Knowledge overconfidence is associated with anti-consensus views

#124

Post by mlyle »

somenameforme wrote: post #123
mlyle wrote: post #122

Earlier quoted context omitted.

I'm getting less and less interested in galloping around talking about different minutiae. I basically disagree with your entire argument. But: > And the climate change example confirms that null hypothesis, which would thus reject your hypothesis. ???! fails to reject the null hypothesis. This seems to be a rather large and fundamental error. You don't ever "confirm[ the] null hypothesis"-- and you certainly don't i…

With all due respect, that feels like a dodge. I decided to look at the result data for this survey [1]. I was unsurprised to find, in a simple skim of only the first survey, at least 2 comments mentioning this issue we're discussing. Asking an e.g. creationist "True/False The Earth is about 4.5 billion years old." and "True/False Scientists claim that the Earth is about 4.5 billion years old." are questions that are…

> With all due respect, that feels like a dodge.

I note you've completely moved on from the huge fallacy of a statistical claim that you made, to choose to circle back to hit one of the other topics we've already discussed earlier.

We all know about the Gish gallop. Pick one. You basically committed data interpretation murder in the grandparent comment.

If I fail to engage with every single point of yours, it doesn't make those points right. Especially when they're repeated after having already been addresses!

> means this is an extremely significant issue.

OK, so there's some questions on the whole test which might inadvertently measure religiosity-- if people don't realize they're supposed to respond with the scientific belief. (Of course, once they're betting on how they will score on that test, it's hard to understand what else they might believe would be the rubric for said test). Even so, this would hardly explains things showing up in the subscores with the same slopes, unless you think the bulk of the test is contaminated.

You're the one pulling the CSVs and running analyses-- is that question missed more by people who have anti-consensus views?

> An interesting issue in itself, but a further problem in this study given relatively few people indicated max dissent, so they're measuring people literally racing through the survey for their $0.85 likely without reading it beyond looking for the "Please select #2 as the answer to this question." questions.

Look, it's just easy to eyeball the graphs and see removing the '7's does nothing to the slope of the lines.

Re: Knowledge overconfidence is associated with anti-consensus views

#125

Post by somenameforme »

mlyle wrote: post #124
somenameforme wrote: post #123

Earlier quoted context omitted.

With all due respect, that feels like a dodge. I decided to look at the result data for this survey [1]. I was unsurprised to find, in a simple skim of only the first survey, at least 2 comments mentioning this issue we're discussing. Asking an e.g. creationist "True/False The Earth is about 4.5 billion years old." and "True/False Scientists claim that the Earth is about 4.5 billion years old." are questions that are…

> With all due respect, that feels like a dodge. I note you've completely moved on from the huge fallacy of a statistical claim that you made, to choose to circle back to hit one of the other topics we've already discussed earlier. We all know about the Gish gallop. Pick one. You basically committed data interpretation murder in the grandparent comment. If I fail to engage with every single point of yours, it doesn't…

The "betting" was for $0.50, which was not lost if they failed. The only cost was not accepting an alternative $0.25 immediate bonus. There's this major issue of sampling only people doing online surveys for pennies a piece, and it is possible that this specific sample might meaningfully change their behavior for $0.50 (or $0.25, depending on how you look at it), but I wouldn't really assume as much. If they really wanted that extra quarter or two, they could have simply looked everything up online.

Looking briefly over the data, anti-consensus views do seem to correlate well with questions that were basically just proxies for measuring that dissent again. For instance one question was 'man was alive at the same time as dinosaurs' type thing. The correct answer on 1-6 is a 1. People who rejected the evolution consensus scored 4.6 on it, with a mode of 7. The mean of all respondents was 3. The score of people who indicated max opposition to e.g. nuclear power was 2.85. Those samples were my first and not cherry picked, beyond intentionally picking a group that would likely have very different biases (opposition to evolution vs opposition to nuclear) to compare against.

Imagine you simply framed the question slightly differently, but testing the exact same knowledge, "Most scientists do not believe humans were alive at the same time as the dinosaurs." I suspect you would have gotten radically different responses. And this issue (in various incarnations) is something I think that is absolutely destroying the 'soft sciences.' Quite small changes in experiment or survey can yield radically different results, and that can be exploited to "prove", more or less, whatever you want to.

p.s. The reason I moved on is because it was simply poor phrasing in a casual anon chat.

Re: Knowledge overconfidence is associated with anti-consensus views

#126

Post by mlyle »

somenameforme wrote: post #125
mlyle wrote: post #124

Earlier quoted context omitted.

> With all due respect, that feels like a dodge. I note you've completely moved on from the huge fallacy of a statistical claim that you made, to choose to circle back to hit one of the other topics we've already discussed earlier. We all know about the Gish gallop. Pick one. You basically committed data interpretation murder in the grandparent comment. If I fail to engage with every single point of yours, it doesn't…

The "betting" was for $0.50, which was not lost if they failed. The only cost was not accepting an alternative $0.25 immediate bonus. There's this major issue of sampling only people doing online surveys for pennies a piece, and it is possible that this specific sample might meaningfully change their behavior for $0.50 (or $0.25, depending on how you look at it), but I wouldn't really assume as much. If they really w…

> p.s. The reason I moved on is because it was simply poor phrasing in a casual anon chat.

It wasn't poor phrasing. Being unable to reject the null hypothesis (on one subtest, even) doesn't mean there's no effect. If you test something 15 different ways, and you get a statistically significant effect with consistent slope on 12 of them, a non-statistically significant effect with a slope in the same direction on 2, and a non-statistically significant effect with a tiny slope in the opposite direction on 1... that looks pretty universal. It's an interesting question for followup research whether those 3 things are actually different in some way, or whether it was luck.

Re: Knowledge overconfidence is associated with anti-consensus views

#127

Post by somenameforme »

mlyle wrote: post #126
somenameforme wrote: post #125

Earlier quoted context omitted.

The "betting" was for $0.50, which was not lost if they failed. The only cost was not accepting an alternative $0.25 immediate bonus. There's this major issue of sampling only people doing online surveys for pennies a piece, and it is possible that this specific sample might meaningfully change their behavior for $0.50 (or $0.25, depending on how you look at it), but I wouldn't really assume as much. If they really w…

> p.s. The reason I moved on is because it was simply poor phrasing in a casual anon chat. It wasn't poor phrasing. Being unable to reject the null hypothesis (on one subtest, even) doesn't mean there's no effect. If you test something 15 different ways, and you get a statistically significant effect with consistent slope on 12 of them, a non-statistically significant effect with a slope in the same direction on 2, a…

You look at a billion and one swans. For the first billion you see nothing but white swans, yet for the final one you look at you see a black swan. You now cannot say swans are universally white. All it takes is a single exception to break a rule.

This is one of the many issues I have with social science - lack of falsifiability. Social science effectively lacks any degree of falsifiability simply because everything is based on vague probabilistic distributions which vary dramatically based on subtle nuances like asking an identical question two different ways, or testing on different populations in cases where the hypothesis is suggested to generalize - as here.

If there was any notion of falsification, it would all be false. In many ways it's quite analogous to astrology which, if you are unaware, was indeed considered a science, studied in universities for hundreds of years. It's end, somewhat ironically, came not from scientific objection to the fact it suffered the same issues as the social sciences today, but from the Church who deemed it heretical in claims of defacto divination.

Re: Knowledge overconfidence is associated with anti-consensus views

#128

Post by mlyle »

somenameforme wrote: post #127
mlyle wrote: post #126

Earlier quoted context omitted.

> p.s. The reason I moved on is because it was simply poor phrasing in a casual anon chat. It wasn't poor phrasing. Being unable to reject the null hypothesis (on one subtest, even) doesn't mean there's no effect. If you test something 15 different ways, and you get a statistically significant effect with consistent slope on 12 of them, a non-statistically significant effect with a slope in the same direction on 2, a…

You look at a billion and one swans. For the first billion you see nothing but white swans, yet for the final one you look at you see a black swan. You now cannot say swans are universally white. All it takes is a single exception to break a rule. This is one of the many issues I have with social science - lack of falsifiability. Social science effectively lacks any degree of falsifiability simply because everything…

> You look at a billion and one swans. For the first billion you see nothing but white swans, yet for the final one you look at you see a black swan.

This analogy doesn't fit our circumstance.

You take two photos each of 7 swans, a close-up and a wide-angle one.

For all of the 7 wide-angle photos, the swan is obviously white.

For the 7 close up-photos: you get 4 good close-up photos of white swan. Then there's 3 bad photos. 2 of them look probably white, and 1 looks like the swan is maybe dark grey.

All the evidence you have is consistent with all the swans being white. Looking to see if you can find those 3 swans and take a better close look at them might be a nice bit of followup.

Re: Knowledge overconfidence is associated with anti-consensus views

#129

Post by somenameforme »

mlyle wrote: post #128
somenameforme wrote: post #127

Earlier quoted context omitted.

You look at a billion and one swans. For the first billion you see nothing but white swans, yet for the final one you look at you see a black swan. You now cannot say swans are universally white. All it takes is a single exception to break a rule. This is one of the many issues I have with social science - lack of falsifiability. Social science effectively lacks any degree of falsifiability simply because everything…

> You look at a billion and one swans. For the first billion you see nothing but white swans, yet for the final one you look at you see a black swan. This analogy doesn't fit our circumstance. You take two photos each of 7 swans, a close-up and a wide-angle one. For all of the 7 wide-angle photos, the swan is obviously white. For the 7 close up-photos: you get 4 good close-up photos of white swan. Then there's 3 bad…

So you have 7 photos, only 4 of them clearly confirm what you want to say, 1 likely contradicting it, and you're happy to take this as a universal? I also think your analogy doesn't hold for another reason. Look at the last study.

This study is intentionally juking their numbers. They "learned" from their first study and removed every single sample that contradicted what they want to 'prove', magnified those that confirmed it, and even under this form of "science" - they still failed to show statistical significance for multiple categories.

It'd be like noticing there were far "dark" swans in Eastland and so going out of your way to make sure in your next safari you limit yourself to Westland. If you genuinely believed all swans were white, and wanted to test that hypothesis, you'd have done the exact opposite. There is only one conclusion to make.

Re: Knowledge overconfidence is associated with anti-consensus views

#130

Post by mlyle »

somenameforme wrote: post #129
mlyle wrote: post #128

Earlier quoted context omitted.

> You look at a billion and one swans. For the first billion you see nothing but white swans, yet for the final one you look at you see a black swan. This analogy doesn't fit our circumstance. You take two photos each of 7 swans, a close-up and a wide-angle one. For all of the 7 wide-angle photos, the swan is obviously white. For the 7 close up-photos: you get 4 good close-up photos of white swan. Then there's 3 bad…

So you have 7 photos, only 4 of them clearly confirm what you want to say, 1 likely contradicting it, and you're happy to take this as a universal? I also think your analogy doesn't hold for another reason. Look at the last study. This study is intentionally juking their numbers. They "learned" from their first study and removed every single sample that contradicted what they want to 'prove', magnified those that con…

> So you have 7 photos, only 4 of them clearly confirm what you want to say, 1 likely contradicting it, and you're happy to take this as a universal? I also think your analogy doesn't hold for another reason

I'm sorry; I disagree and I don't think you understand the statistics.

> Look at the last study.

This is another new goalpost. I am not willing to discuss further.

Post reply on HN