Live data from Hacker News

A researcher's response to people pointing out 150 errors in four of his papers

andrewgelman.com

21–30 of 105 posts

Re: A researcher's response to people pointing out 150 errors in four of his papers

#21
post #15
post #10

Earlier quoted context omitted.

I've found I'm exceptional in being at all concerned about privacy. I think that if you asked, most people would only want minimal effort put towards confidentiality, especially since the trade-off is that other researchers won't be able to double check the analysis.

I've worked as a journalist in which my (and many other data journalists') modus operandi was to publish the data. Often because the data was public record anyway. I now work in academia and the mentality is significantly different. Some of it is logistics -- I would say most traditional news organizations do not have the internal incentive or habit to figure out a way to publish data. Whereas with newer organization…

To add a bit more for people who haven't been involved in an IRB:

The strictness varies by university. Ultimately the IRB is there to ensure safety of the participants and minimize the chance of negative consequences. A couple examples I've heard from around my university include...

A study was taking place outside. There was a chance of a bee sting occurring to the participant. The IRB required that the researcher have an epi-pen ready just in case along with any required training.

A paper used in a study had the wrong stamp on it (not the most recent IRB stamp). The document had not changed from the last time, but the rule was simple. Every document presented to the participant had to have the most recent stamp.

On a study in which it was expected that 1/2 of the participants would feel nauseous, they had to provide a place to sit, water, and small snacks.

And perhaps most importantly, you can quit any study you're participating in at any time, for any reason. Compensation is figured out ahead of time, and you're going to get lots of questions from IRB if you want to do anything to reduce the compensation for leaving part way through the experiment.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#22
post #4

I still don't understand: Is there any good reason that raw data should not be published along with each paper?

There are sometimes ethics rules on how long data is stored for. If there's a commitment to destroy the raw data after a period, eg. five years, often it can't be publicly released (even anonymised) or there'd be no way of ensuring it's been destroyed when the time is up.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#23

"except that this sort of ridiculous hype is standard operating procedure among celebrated and leading researchers in psychology and economics." Thank god it's not SOP among actual sciences.

I hope you're being sarcastic. I'm a bit hesitant to provide examples, but a lot of top results in computer science are sold as being much more far reaching than what they really are.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#24
A sidenote:

Using a snappy XKCD is not a valid logical argument even though the poster might think it makes one look hip and cool. I wish researchers were not this lazy.

There are at least a handful of valid arguments that one could use rather than an XKCD cartoon.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#25
post #3
post #2

I imagine Wansink does get it. Now that he's outed, he either trolling for fun, or hoping the blasé approach is the one that quiets all the fuss the fastest.

I wouldn't be certain. A lot of researchers really are clueless about anything to do with statistics. It takes much more work/time to do real research, so the funding system has actually been selecting for people who can remain ignorant (so it is not fraud) and just produce p-hacked "results". In many areas, this has been going on for multiple generations now and you are trained to do it as a grad student.

> I wouldn't be certain. A lot of researchers really are clueless about anything to do with statistics.

This is definitely true. I once saw a book with a title similar to "Statistics for Dummies" in a professors office. He had plenty of access to staticians at the university too. Unfortunately if a given field involves many people with ignorance of statistics, these problems may not be called out during the peer review process.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#26
post #17

It's weird. If conservatives don't believe scientists, it's because the conservatives are labelled "morons". Could it be that conservatives just think there's a lot more of this crappy "science" out there, and that a situation like this is the canary in the coal mine? When presented with examples like this, it's actually _irrational_ to think that there aren't other egregious papers out there. Possibly just a few, bu…

Of course there are lots. There's a bucket-load of papers get published every year, and sometimes reviewers are in a hurry too and miss things. A miniscule number of papers ever replicated, and replication studies regularly find reproducability rates of less than 50%. Fortunately, critical thought does not stop at the point of publication. When researchers read each other's papers they (hopefully) do not blindly assu…

I think there's a general popular misunderstanding of the process of science as it is generally practiced.

A single paper isn't as meaningful as you might think. It's more like an interesting blog post. You might use it as a starting point for your own investigation.

A single paper does not represent "the truth" as we currently understand it. For that you need to have a broad scientific consensus.

You might be able to get a better understanding of that from review papers citing recent advances in the field. Or from attending conferences/reading a broad range of papers yourself.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#27
post #9
post #8

Earlier quoted context omitted.

Why should it only matter if a substantial number of subjects care about confidentiality? The danger for some people is death or worse. Why are you dismissing that risk?

you got the question backwards. if enough subjects don't care about confidentiality you could use only them and publish the data.

Doing so would be publishing non-representative data.

Imagine an identified versus anonymous survey about cocaine usage in America. People will likely answer honestly if it is anonymous.

If it is identified, most people would be fine with it being identified and would mark no usage. People who do use cocaine would be more likely to not participate or lie on the answer.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#28
post #17

It's weird. If conservatives don't believe scientists, it's because the conservatives are labelled "morons". Could it be that conservatives just think there's a lot more of this crappy "science" out there, and that a situation like this is the canary in the coal mine? When presented with examples like this, it's actually _irrational_ to think that there aren't other egregious papers out there. Possibly just a few, bu…

My thought: ridiculous false equivalency.

Conservatives don't distrust science carte blanc. Many of them will chuck money at asinine bullshit as long as it's billed as a way to kill the other team. Check out some of the more ridiculous stuff to come out of DoD.

They distrust climate science. And they don't distrust the actual science, just the conclusions. And increasingly not even the most basic conclusion, just the most obvious causal theory behind that conclusion.

Let's not throw the baby out with the bath water.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#29
post #17

It's weird. If conservatives don't believe scientists, it's because the conservatives are labelled "morons". Could it be that conservatives just think there's a lot more of this crappy "science" out there, and that a situation like this is the canary in the coal mine? When presented with examples like this, it's actually _irrational_ to think that there aren't other egregious papers out there. Possibly just a few, bu…

I've had this same conversation with colleagues. I think there's a mixture of things going on.

There's a lot more of this going on in a lot of fields, especially the biomedical and social sciences, but probably everywhere.

I think conservatives are correct to pick up on this general phenomenon. They know there's a lot of systemic problems with research and academics, and approach things from that perspective.

The problem, to me, is that they then overstep and use that to justify an equally pernicious position, which is to denounce or dismiss science in general, rather than to encourage fixes or constructive change. Of course, depending on the conservative, they're more likely denounce scientific research on certain topics and areas of research than others.

It's throwing baby out with bathwater, basically.

The scientific community then reacts poorly, in my opinion, by denying the kernel of truth there is to their concerns, and not addressing them.

You can see this playing out in some conservative blogs. They're aware of the replication crisis, discuss it, and point to things like the email leaks by climate scientists, which in my memory were scientifically sort of moot, but did reveal how much scientists are very much human with human flaws and motivations.

Maybe I'm wrong, though, and the typical response from the scientific community is best, though: maybe it's better to treat the problems with science separate from the conservative attacks, to tackle one problem at a time.

I guess I see two problems, one being problems with academics and science, the other being the involvement of politicians in science. I think the second thing only makes the first problem worse. It's not like the politicians aren't serving their own unholy masters.

Re: A researcher's response to people pointing out 150 errors in four of his papers

#30

"except that this sort of ridiculous hype is standard operating procedure among celebrated and leading researchers in psychology and economics." Thank god it's not SOP among actual sciences.

I hope you're being sarcastic. I'm a bit hesitant to provide examples, but a lot of top results in computer science are sold as being much more far reaching than what they really are.

Speaking from my own areas of experience, I've been struck by how much some celebrated methods and results in machine learning, broadly speaking, basically amount to tinkering without any hard generalizable proofs. In at least one case I'm aware of, there was a statistical proof later that made it clear the original researchers were close, but if they had a different dataset they would have come up with a different result. In fact, as I write this, I think I recall that different researchers did come up with different results, and it was in large part due to tinkering on different datasets.

So yes, even in computer science you run into similar things. Maybe not the same, but similar processes leading to similar problems.

Post reply on HN