Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Noted study in psychology fails to replicate, crumbles with evidence of fraud

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

31–40 of 110 posts

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#31

Post by parliowl »

wanderingmind wrote: post #4
zhdc1 wrote: post #2

I'm pretty sure this was discussed in an earlier HN post. However, I want to point out that "not replicating" doesn't necessarily equate to bad science (or, more accurately, bad practice on the part of the principal investigators irt accepted best practices in their field). There are a bunch of studies in the social sciences that failed to replicate simply because a p value was slightly below a threshold target, even…

Failure to replicate is the definition of bad science. Science is based on objective, repeatable experiments. If the experiments can't even be reproduced how can they be hypothesized and validated?

This is so not true on many levels. Science is not about being right every time. Positive experimental findings can turn out to be wrong even if the study was excellent. This is called a type 1 error, and indeed they are guaranteed to happen with 5% probability assuming every study is conducted correctly and people don’t cheat.

Also, I don’t think you understand that there is a difference between “replicate” and “reproduce” as those terms are typically used in social and biological science. Reproduction refers to the methods and is simply conducting the experiment again, regardless of the outcome. An experiment should be reproducible, of course—if it isn’t, the paper was poorly written (the experiment might still be ok though).

Replication refers to the results: do we get the same answers as the original experiment? Bad science is one way an experiment might not replicate but other reasons are 1. Just a regular type 1 error in the original study, 2. A type 2 error in the new one, 3. an overlooked methodological difference or mistake somewhere. No. 3 doesn’t mean bad science, it could be just normal scientists making normal mistakes, as they sometimes do.

Also, you don’t “hypothesize” an experiment. You come up with a hypothesis then test it with an experiment. (Sometimes scientists HARK but that’s a different discussion …)

In this case, it sounds like there truly was some cheating going on of the worst kind (data fabrication).

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#32

Post by MereInterest »

benmanns wrote: post #9
wanderingmind wrote: post #4

Earlier quoted context omitted.

Failure to replicate is the definition of bad science. Science is based on objective, repeatable experiments. If the experiments can't even be reproduced how can they be hypothesized and validated?

If we report p values around 0.05 wouldn’t you expect about 1 in 20 studies failing to replicate without being bad science?

Not quite. That's the common misinterpretation of p-values as probabilities that a study is correct. The p-value isn't the probability of the hypothesis being wrong given the data. It is instead the probability of the data given the hypothesis being wrong.

In order to determine what proportion of studies would fail to reproduce, you would also need to know the number of correct hypotheses in the first place. If 100 correct hypotheses and 900 incorrect hypotheses are tested, your sample of "pThis is why there is such a large push for prepublishing of analysis methods, hypotheses, and experimental design, and later publishing of negative results. The initial distribution of hypotheses is important to know how to interpret the resulting body of work.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#33

Post by smitty1e »

newsclues wrote: post #13
willis936 wrote: post #8

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits…

Publishing negative results is important to solving this issue.

Incentivizing publishing negative results is a challenge.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#34

Post by auganov »

If top scientists are willing to outright manufacture data, how many would be willing to intentionally follow unsound logic? As in dishonestly claiming some (legitimate) data implies something that it doesn't or intentionally avoiding alternative explanations. You'd have to assume there's even more of that and all other kinds of soft fraud going on.

Ideally planning data production, execution of data production and analysis would all be done by different people and published separately. At the very least you'd want total transparency at each stage.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#35

Post by gus_massa »

willis936 wrote: post #8

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits…

> The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results.

One important task of the advisor is to filter bad projects. Does the paper pass the smell test? Are the authors reliable or they had published "dubious" results before?

[I was recently discussing a paper in a virtual meeting. My conclusion is that Zoom does not have enough negative reactions.]

Also, before assigning an experiment to a graduate student, try to have something like a plan:

1) This will surely work and is interesting enough for the dissertation and to be publish.

2) This part is dubious, because it's related to a too resent research paper or there may be some hidden technical problems that are difficult to see.

2') An underspecified alternative to 2, just in case.

3) Extra task for bonus points and perhaps a Nobel price.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#36

Post by dalbasal »

profstasiak wrote: post #14

I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility cri…

I think you need to draw a line between "not replicable" and fraudulent.

This is about the latter. This is the kind of corruption that spreads. It's a very bad thing.

It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the scientific method where they don't work well.

Say you are studying the effects of a breakfast program, 2 hr recess or push-up regiment on high schoolers. The study is well constructed, faithfully conducted, etc. You find interesting, potentially actionable ways of reducing obesity, improving psych health or something.

Do/Should we even expect such a study to replicate in 1990s Tokyo, 2000s Miami, School A, School B, etc.

Does that mean this study should not be conducted? Does it mean we can't generate knowledge and understanding of things this way? Is there any daylight between "anecdote" and a result that qualifies for publication in Physics?

A lot of articles published in many fields feel like they are being hammered into a Popperian mold they don't fit into. Everyone knows it, but no one is willing to say "this method doesn't work here. we're actually doing a different thing." Instead, they sort of eye roll the article into the frame and let readers unscramble it for themselves.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#37

Post by detaro »

willis936 wrote: post #16
newsclues wrote: post #13

Earlier quoted context omitted.

Publishing negative results is important to solving this issue.

I agree. The grad students agree. Now how do you convince a group obsessed with career advancement and holding all of the cards (PIs) to agree?

By convincing the people deciding about their career advancement that negative results are worth something too. (no, that isn't easy, since it's the same people to a good degree, which is a problem with solving many problems in academia)

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#38

Post by unityByFreedom »

willis936 wrote: post #16
newsclues wrote: post #13

Earlier quoted context omitted.

Publishing negative results is important to solving this issue.

I agree. The grad students agree. Now how do you convince a group obsessed with career advancement and holding all of the cards (PIs) to agree?

Enough of these "falsified science" stories come out and people will become interested in stories debunking bad science. I subscribe to a podcast called "everything hertz" for this reason.

https://everythinghertz.com/

https://twitter.com/jamesheathers

https://twitter.com/dsquintana

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#39

Post by rwbhn »

dalbasal wrote: post #36
profstasiak wrote: post #14

I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility cri…

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the…

If the results are not replicable, how can they be actionable?

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#40

Post by throw_m239339 »

ghyiihdruh wrote: post #20

What's the purpose of this? It will just give ammo to anti-science Trump like people. It's harmful. This kind of stuff should be handled as quietly as possible, and definitely not in public. Science is already under attack, we need not undermine it from the inside too.

> What's the purpose of this? It will just give ammo to anti-science Trump like people.

It's bizarre how today, in USA, for some it's all about "the other camp can't have a win, at any cost", and not what is right. Fake science hurts science the most, and covering it up IS WHAT is going to give ammo to whoever you do not like, calling a fraud a fraud doesn't do that.

Post reply on HN