An erroneous paper on religion and generosity is retracted • Hacker News Board

Re: An erroneous paper on religion and generosity is retracted

#21

Post by gumby » Fri, Sep 27, 2019, 7:16 PM UTC

astine wrote: post #7

charliesharding wrote: post #6

Earlier quoted context omitted.

Really sad to see and feels like it's becoming more common (maybe just because I'm paying closer attention). If it fits the narrative, accept first, retract later. It would be interesting to see view statistics on the original article vs the retraction.

It's not just narrative fitting. There is also a strong bias towards publishing results that seem surprising because that gets more readers. Of course, that also biases toward wrong results because wrong results are likely to be surprising.

"wrong results are [more likely] to be surprising".

This case is interesting because there's a large population who would find these (unproven as it turns out) results confirmatory rather than unexpected.

In the end they were neither.

Re: An erroneous paper on religion and generosity is retracted

#22

Post by joelhoffman » Fri, Sep 27, 2019, 7:17 PM UTC

UncleMeat wrote: post #11

shriphani wrote: post #10

From the article: Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1). I mean I don't even understand how this seemed like a normal thing to do?

You've never had a bug in your code that seemed insane after the fact?

This is why we have code review processes. It's long past time for that to be part of formal scientific peer review.

Re: An erroneous paper on religion and generosity is retracted

#23

Post by makomk » Fri, Sep 27, 2019, 7:20 PM UTC

Alas, these kinds of problems are not restricted to the social sciences. Case in point, this retraction from a couple of days ago: https://retractionwatch.com/2019/09/25/nature-paper-on-ocean... Very similar to this one really; the paper claimed to overturn our existing knowledge in a way that fitted a narrative people were inclined to believe (in that case: we're all doomed) and was immediately seized on by all the news sites because of it, except the statistics were mucked up and it couldn't show what it claimed to. The fact that it was so surprising should've been even more of a massive warning sign in that case though.

Re: An erroneous paper on religion and generosity is retracted

#24

Post by lmkg » Fri, Sep 27, 2019, 7:21 PM UTC

shriphani wrote: post #10

From the article: Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1). I mean I don't even understand how this seemed like a normal thing to do?

The variable for Country should have been treated as a categorical variable, but was instead processed as a numeric variable.

This mistake would be downright trivial to make in R. Just declare that Country is a Factor (which is the built-in type for categorical variables), and then throw the data into a library whose attitude towards errors is to coerce everything to numbers until the warnings go away.

Background: Factors in R are the idiomatic way to work with categorical data, and they work somewhat like C-style enums except the variants come from the data rather than a declaration. So if you take a column of strings in a data frame and cast it to a Factor, it will generate a mapping where the first distinct value is coded as 1, the second distinct value is coded as 2, etc. Then it replaces the strings with their integer equivalents, and saves the mapping off to the side.

I forget the exact rules (if there are rules, R is a bit lawless), but it's not very hard to peek under the hood at the underlying numeric representation. Many built-in operations "know" that Factors are different (e.g. regressing against a Factor will create dummy variables for each variant), but it's up to each library author how 'clever' they want to be.

Re: An erroneous paper on religion and generosity is retracted

#25

Post by not2b » Fri, Sep 27, 2019, 7:21 PM UTC

xibalba wrote: post #19

shriphani wrote: post #10

From the article: Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1). I mean I don't even understand how this seemed like a normal thing to do?

Agreed, but maybe you have to assume that the scientist knows very little about coding for data science, which is effectively what we're talking about here. I think a major contributing factor to problems like this is people going into the soft/social sciences being more likely to be math/stats AND programming averse. Meanwhile, all sciences continue their long term trend towards applied math via programming. This le…

Social sciences programs require students to take statistics courses. That's no guarantee that statistics will be correctly applied.

Re: An erroneous paper on religion and generosity is retracted

#26

Post by human20190310 » Fri, Sep 27, 2019, 7:21 PM UTC

leereeves wrote: post #15

shriphani wrote: post #10

From the article: Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1). I mean I don't even understand how this seemed like a normal thing to do?

It's really very easy to do, roughly the stats equivalent of declaring a variable signed instead of unsigned. Many algorithms work on both categorical and continuous variables, with different results depending on the variable's type.

At risk of embarrassing my self statistically, what exactly happens when you do this?

I.e., if you're controlling for country, that means you're bucketing by country, and looking at each subset, right? So if country is represented by a non-discrete value... what exactly happens?

Re: An erroneous paper on religion and generosity is retracted

#27

Post by whymsicalburito » Fri, Sep 27, 2019, 7:23 PM UTC

There were even news sites that published articles about the original article AFTER the retraction was announced! The state of science reporting is very sad.

Re: An erroneous paper on religion and generosity is retracted

#28

Post by m0zg » Fri, Sep 27, 2019, 7:24 PM UTC

Another, more consequential paper that popped up on HN recently has been retracted as well. The one about ocean warming. The retraction notice is a masterclass on weasel language, worth reading in its own right.

https://www.nature.com/articles/s41586-019-1585-5

Re: An erroneous paper on religion and generosity is retracted

#29

Post by acbart » Fri, Sep 27, 2019, 7:25 PM UTC

not2b wrote: post #25

xibalba wrote: post #19

Earlier quoted context omitted.

Agreed, but maybe you have to assume that the scientist knows very little about coding for data science, which is effectively what we're talking about here. I think a major contributing factor to problems like this is people going into the soft/social sciences being more likely to be math/stats AND programming averse. Meanwhile, all sciences continue their long term trend towards applied math via programming. This le…

Social sciences programs require students to take statistics courses. That's no guarantee that statistics will be correctly applied.

Or that the material will be taught effectively. Or that the students are contextually prepared to understand the material at that point. Or any of a number of other things that go wrong when people suggest that education is the solution/root to a very hard problem :)

Re: An erroneous paper on religion and generosity is retracted

#30

Post by kachnuv_ocasek » Fri, Sep 27, 2019, 7:28 PM UTC

human20190310 wrote: post #26

leereeves wrote: post #15

Earlier quoted context omitted.

It's really very easy to do, roughly the stats equivalent of declaring a variable signed instead of unsigned. Many algorithms work on both categorical and continuous variables, with different results depending on the variable's type.

At risk of embarrassing my self statistically, what exactly happens when you do this? I.e., if you're controlling for country, that means you're bucketing by country, and looking at each subset, right? So if country is represented by a non-discrete value... what exactly happens?

In short, ANOVA is usually what you want to do: https://en.wikipedia.org/wiki/One-way_analysis_of_variance

In practice, if you have n countries, you'll add n-1 binary variables to your regression equation. The first country is the reference level (all zeros), for the second country set the first new variable to one, the rest to zero, etc.