Earlier quoted context omitted.
You've never had a bug in your code that seemed insane after the fact?
This is why we have code review processes. It's long past time for that to be part of formal scientific peer review.
[deleted]
An erroneous paper on religion and generosity is retracted
31–40 of 165 posts
Re: An erroneous paper on religion and generosity is retracted
#32"But when they included their categorically-coded country (1 = US, 2 = Canada, and so on) in their models, it was entered not as fixed effects, with dummy variables for all of the countries except one, but as a continuous measure. This treats the variable as a measure of ‘country-ness’ (for example, Canada is twice as much a country as the US) instead of providing the fixed effects they explicitly intended" How did t…
I don't think it's quite as obvious of an error as you are suggesting.
They were trying to correct for scenarios like this: Hypothetically, Canadians are twice as generous as Americans and twice as religious, but religious Canadians are equally generous as non-religious Canadians and religious Americans are equally generous as non-religious Americans. On the surface, it appears that religious people are more generous, but really it's just that Canadians are more generous.
Instead of treating the countries as discrete groupings, they treated them as points on a spectrum with each country being assigned an arbitrary place on the spectrum.
If #3 happened to be China, they would be assuming that people in China should very similar to people in the US and Canada, because 1 vs. 3 on a scale that goes to 200 is hardly any difference at all, but really the numbers are just arbitrary identifiers.
Re: An erroneous paper on religion and generosity is retracted
#33From the article: Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1). I mean I don't even understand how this seemed like a normal thing to do?
The variable for Country should have been treated as a categorical variable, but was instead processed as a numeric variable. This mistake would be downright trivial to make in R. Just declare that Country is a Factor (which is the built-in type for categorical variables), and then throw the data into a library whose attitude towards errors is to coerce everything to numbers until the warnings go away. Background: Fa…
This makes the most sense to me. I don't work in the dataframes world but without this explanation it seemed like someone would have to go out of their way to make that error.
Re: An erroneous paper on religion and generosity is retracted
#34Sometimes I feel weird coding zip codes as strings but this is a great example why. If my program ever treats a zip code like a number I would like it to throw an error. At least in this case the error looks like an accident.
On topic, from yesterday: https://news.ycombinator.com/item?id=21067764
It's another social sciences paper but in this case a co-author has requested a retraction over his strong belief that the paper includes fabricated data. The retraction request has been denied. It differs from this paper in that the data anomalies look intentional.
Re: An erroneous paper on religion and generosity is retracted
#35Re: An erroneous paper on religion and generosity is retracted
#36Sometimes I feel weird coding zip codes as strings but this is a great example why. If my program ever treats a zip code like a number I would like it to throw an error. At least in this case the error looks like an accident. On topic, from yesterday: https://news.ycombinator.com/item?id=21067764 It's another social sciences paper but in this case a co-author has requested a retraction over his strong belief that the…
Non-US postal codes often include letters and spaces, and some areas don't use postal codes at all.
Re: An erroneous paper on religion and generosity is retracted
#37Sometimes I feel weird coding zip codes as strings but this is a great example why. If my program ever treats a zip code like a number I would like it to throw an error. At least in this case the error looks like an accident. On topic, from yesterday: https://news.ycombinator.com/item?id=21067764 It's another social sciences paper but in this case a co-author has requested a retraction over his strong belief that the…
My rule of thumb is that anything that’s not part of a calculation will be a string. Any chance I ever want to multiply a zip code? Doubtful. String, it is.
Re: An erroneous paper on religion and generosity is retracted
#38Earlier quoted context omitted.
At risk of embarrassing my self statistically, what exactly happens when you do this? I.e., if you're controlling for country, that means you're bucketing by country, and looking at each subset, right? So if country is represented by a non-discrete value... what exactly happens?
In short, ANOVA is usually what you want to do: https://en.wikipedia.org/wiki/One-way_analysis_of_variance In practice, if you have n countries, you'll add n-1 binary variables to your regression equation. The first country is the reference level (all zeros), for the second country set the first new variable to one, the rest to zero, etc.
So one-hot encoding, plus one "none-hot" base case.
Why not just one-hot for all? To save one input?
Re: An erroneous paper on religion and generosity is retracted
#39From the article: Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1). I mean I don't even understand how this seemed like a normal thing to do?
Simple...data representation is not the same as data meaning.
I teach an introductory stats course and we hammer this in. Categorical data are often represented as numbers or other short indicators for storage purposes. Typically I fmultiple choice the encoding is by the order of the choice options.
I not infrequently see average of gender because male = 0 and female = 1 or vice versa and someone generates a table without thinking.
Re: An erroneous paper on religion and generosity is retracted
#40"The paper received a great deal of attention, and was covered by over 80 media outlets including The Economist, the Boston Globe, the Los Angeles Times, and Scientific American." And how many of these will cover the retraction? A dozen at most? And all those articles will be sitting out there, getting cited and read on occasion.
Really sad to see and feels like it's becoming more common (maybe just because I'm paying closer attention). If it fits the narrative, accept first, retract later. It would be interesting to see view statistics on the original article vs the retraction.
It's not sad, it's great.
The history of science is full of drama where most issues took multiple generations to resolve. It's easy to forget, in those interim periods, people would build all kinds of castles on total BS all the time that cost society in so many ways.
Today stuff gets resolved faster and that's a good thing. People, qualified or not, who get carried away by hype or bias look foolish much much faster. And thanks to how hard it is to erase mistakes from the internet good luck rebuilding lost cred.