Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A researcher's response to people pointing out 150 errors in four of his papers

andrewgelman.com

Post reply on HN
Print view View original

11–20 of 105 posts

Re: A researcher's response to people pointing out 150 errors in four of his papers

#11

Post by wfunction »

nonbel wrote: post #10
wfunction wrote: post #7

Earlier quoted context omitted.

> Does a substantial portion of the subjects actually care about this? I don't think that's how laws or ethics rules work, for one. And I wasn't talking about a particular experiment, for two.

I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis.

> I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis.

Like I said, that's not how laws work.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#12

Post by froindt »

Andys wrote: post #4

I still don't understand: Is there any good reason that raw data should not be published along with each paper?

It depends on the situation. I work with a professor who came from industry. He once worked on a project where they did massive data collection inside the military. Over $1 million was spent over 2 weeks collecting data. A couple months after, some researcher called and said "Hey, I heard about the data you collected. Would you mind passing it on to me?". Of course the answer was no, not until we publish damn near everything we can out of it.

Another consideration is the amount of data. Sometimes you're talking about dozens or hundreds of gigabytes of data. Start multiplying this by the scale of universities and you come up with a large expense to store massive amounts of data most of which will be downloaded no more than a couple times. On top of that, the data must be cleaned and made anonymous, and if they make a mistake in the process, there could be large liability issues. See AOL https://en.wikipedia.org/wiki/AOL_search_data_leak

Re: A researcher's response to people pointing out 150 errors in four of his papers

#14

Post by froindt »

nonbel wrote: post #6
wfunction wrote: post #5

Earlier quoted context omitted.

> I still don't understand: Is there any good reason that raw data should not be published along with each paper? Confidentiality? What if correlations give away subjects' identities?

Does a substantial portion of the subjects actually care about this?

Even if they don't care, there is a responsibility by the researchers to the participants to protect their identity. Your participation in a study may have no big effect right now, but down the line it could. If your project has federal funding and involves research on people, there are ethical requirements and trainings required before starting the study.

What if you completed a simple survey for a study and identified yourself as a member of a group (Christian, Jewish, Mexican, African American, etc.) and 5 years down the line, there was a big rounding up of your group. Suddenly your participation 5 years ago is leading to terrible life changes.

Please get to know participant protections more. They are very important for the integrity of the data used in academia. The NIH has a strict policies to ensure the safety of participants.

The Institutional Review Board (IRB) reviews studies for risks participants could face as a result of participating. Every person who has access to the de-anonymized data must complete an NIH training course.

Depending on your institution, the process varies in seriousness. Having worked on 2 submissions and helped on ~3 other IRB approved studies, it can be tedious. My institution takes 6+ weeks from submission to approval, which can really delay things. At the same time, it's a vital part of the process to ensure validity.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#15

Post by danso »

nonbel wrote: post #10
wfunction wrote: post #7

Earlier quoted context omitted.

> Does a substantial portion of the subjects actually care about this? I don't think that's how laws or ethics rules work, for one. And I wasn't talking about a particular experiment, for two.

I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis.

I've worked as a journalist in which my (and many other data journalists') modus operandi was to publish the data. Often because the data was public record anyway. I now work in academia and the mentality is significantly different. Some of it is logistics -- I would say most traditional news organizations do not have the internal incentive or habit to figure out a way to publish data. Whereas with newer organizations, such as 538 [0] and Buzzfeed News [1], the data teams have editors to whom open-source and digital publishing is more the natural way of things.

But in academia, there are also set rules and precautions governing every study. I haven't proposed any research yet but my understanding is that if your study requires collecting data from participants, the Institutional Review Board requires you to be very clear to participants about privacy and confidentiality and that you follow the guidelines to the letter.

Additionally, there are datasets only available to academics that aren't available to non-academics (i.e. journalists), which speaks to the expectation that academics be very mindful about confidentiality promises.

[0] https://github.com/fivethirtyeight/data

[1] https://github.com/BuzzFeedNews/everything

Re: A researcher's response to people pointing out 150 errors in four of his papers

#16

Post by cs2818 »

Andys wrote: post #4

I still don't understand: Is there any good reason that raw data should not be published along with each paper?

In some of my research the participants are young children or are disclosing sensitive social information during the study, so we typically cannot share the data in fine-grained detail. It can be difficult to ensure confidentiality will be maintained for the participants.

I usually do my best to find a subset of variables that can be released in an anonymized form and still capture the true essence of the results and make that available. This isn't always easy to do though depending on the study details.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#17

Post by MR4D »

It's weird. If conservatives don't believe scientists, it's because the conservatives are labelled "morons". Could it be that conservatives just think there's a lot more of this crappy "science" out there, and that a situation like this is the canary in the coal mine?

When presented with examples like this, it's actually _irrational_ to think that there aren't other egregious papers out there. Possibly just a few, but possibly a large number of them.

Thoughts?

Re: A researcher's response to people pointing out 150 errors in four of his papers

#18

Post by froindt »

nonbel wrote: post #10
wfunction wrote: post #7

Earlier quoted context omitted.

> Does a substantial portion of the subjects actually care about this? I don't think that's how laws or ethics rules work, for one. And I wasn't talking about a particular experiment, for two.

I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis.

There is also the question of how honest people will be when they are anonymous versus not. If you were studying cocaine use in America, publishing your identified data (to enable double-checking the analysis) would give the cops a list of people who admitted to illegal drug usage.

The current system with IRB allows for the collection of identified data. It also allows the data to be released, in full, as long as there is no way for an individual to be identified specifically. In some cases you can simply swap names for a participant ID (after randomly sorting the list). In other cases you have no choice but to publish some summary statistics because the data you collected is only applicable to a very small number of people in a small region, or there is enough data people could reverse-engineer likely participants of the study.

Science is about self-correcting when errors are made. There are plenty of reasons to criticize the current incentive structure for academics, such as there being little funding to replicate other studies, but doing away with confidentiality is absolutely not the way to go about it.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#19

Post by a_bonobo »

Andys wrote: post #4

I still don't understand: Is there any good reason that raw data should not be published along with each paper?

An old concern is that someone comes along, takes your data, and then

a) shows that you have an error somewhere and disproves your paper, making you look stupid, or

b) does something better with your data than your paper, making you look stupid, or

c) runs a follow up study on the same data you are currently running and publishes before you have the chance

In my experience usually none of this happens, UNLESS you're in a really cut-throat/hotly debated/bleeding edge corner of science, which is fairly rare.

c) is actually problematic if you told your funding body that you're going to generate X studies with the data they paid for, since you can only publish X-1 studies. I've seen this happen with biologists not releasing their genome assemblies with the assembly paper so they could run basic follow up studies using that assembly as reference before anyone else can do it.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#20

Post by wbillingsley »

MR4D wrote: post #17

It's weird. If conservatives don't believe scientists, it's because the conservatives are labelled "morons". Could it be that conservatives just think there's a lot more of this crappy "science" out there, and that a situation like this is the canary in the coal mine? When presented with examples like this, it's actually _irrational_ to think that there aren't other egregious papers out there. Possibly just a few, bu…

Of course there are lots. There's a bucket-load of papers get published every year, and sometimes reviewers are in a hurry too and miss things. A miniscule number of papers ever replicated, and replication studies regularly find reproducability rates of less than 50%.

Fortunately, critical thought does not stop at the point of publication. When researchers read each other's papers they (hopefully) do not blindly assume them to be bug-free.

Post reply on HN