Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Noted study in psychology fails to replicate, crumbles with evidence of fraud

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#81

Post by dalbasal »

nyolfen wrote: post #74
dalbasal wrote: post #36

Earlier quoted context omitted.

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the…

the replication crisis came to a head a few years ago when it was revealed that two thirds of social psychology findings failed to replicate -- this is not just empiricism taken too far, this is the mass production of false knowledge due to poor training at best or incentive misalignment at worst

True, and a lot of it is fraud or not far.

However, a big chunk of this statistic is not actually fraud. It's just false empiricism. If you look at a lot of the actual papers, I think you'll find that a lot of them just aren't trying to do empiricism... at least not strict empiricism. They're forced into that structure by a convention.

Either (1) "Not Science" becomes OK and extends to economics, some areas of ecology, and such. (2) The Scientific Method gets renamed "Method A," and the concept of scientific pursuit is more flexible, depending on what you are studying or (3) We force these poor bastards doing economics, psychology and such to pretend that they're doing a method that they're not... or publish in the prestigious "Unscientific Economics & Psychology" journal.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#82

Post by JumpCrisscross »

mountainb wrote: post #60
willis936 wrote: post #8

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits…

In other fields where there is a chance for fraud, the honor system is not used. In academia, the attitude is just 'the honor system is good enough, WINK' as it relates to data. In other fields where there are opportunities for embezzlement or other abuses of trust everything is monitored, only some employees are authorized to do certain transactions, and many of those employees are bound by law and oath to not do th…

> With this abuse of trust, the damage is more to the integrity of the purportedly rational knowledge base

It corrodes trust in our scientific institutions. When that pop something talk by an acclaimed psychologist turns out to be not only bullshit but outright fraud, it’s not a huge leap to project that mistrust onto e.g. the medical establishment or climate scientists.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#83

Post by rossdavidh »

dalbasal wrote: post #36
profstasiak wrote: post #14

I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility cri…

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the…

I think the current situation calls for a more rigorous application of Popperian standards, not less.

Nothing about "this study applies to this specific situation" is counter to Popper. However, if it fails to replicate in other situations, it tends to suggest at the very least "might be too specific to this situation to have practical application".

This happens in software as well. If my system works on my specific laptop but anyone with a slightly different version of the programming language, imported libraries, or operating system will be unlikely to get it to work, then it's just not good software. The solution is not (usually) to put something in the README that says "will only work with python 3.6.7 on Ubuntu 18.04", the solution is to change the software so it is more generally applicable. And it's on me the programmer, to make that happen. If the scientific phenomenon is specific to 1990's Tokyo but not 2000s Miami, then some major variable is being left out, and the science requires finding out what that is.

I agree with your basic point that there is a big difference between non-replicable and fraud, though.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#84

Post by sennight »

mensetmanusman wrote: post #69

What if humans were generally so unique that there were no useful generalizations in psychology. (E.g every attempt at an experiment like this was just noise) How would that affect the field?

It would probably focus on measuring the degree to which individuals differ, assigning explanatory variables, and figuring out the theoretical maximum (as brain chemistry sets a pretty hard limit). But that doesn't really matter, because we know that people can very easily be generalized - there is an entire industry devoted to it: advertising. Many years ago I had to design a process to pre-stage inventory based on expected demand - using historical data from a fairly inelastic customer base. It was spooky how accurately a Poisson distribution could predict long tailed events - but only after segmenting the customer base into something like 5 groups. This was for millions of people - 5 groups.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#85

Post by matthewdgreen »

mountainb wrote: post #60
willis936 wrote: post #8

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits…

In other fields where there is a chance for fraud, the honor system is not used. In academia, the attitude is just 'the honor system is good enough, WINK' as it relates to data. In other fields where there are opportunities for embezzlement or other abuses of trust everything is monitored, only some employees are authorized to do certain transactions, and many of those employees are bound by law and oath to not do th…

Everything is a cost/benefit tradeoff. We devote more resources to securing your bank account against fraud because we (as a society) think that it’s worth spending the money — on the theory that recovering your money after it’s stolen is very hard and because we think it’s acceptable to cause legitimate transactions to undergo some inconvenience. We devote fewer resources to verifying every social sciences experiment because society doesn’t want to spend the money to reduce a presumably already-low fraud rate and because the worst case outcome is usually articles like this one. And yet despite these decisions, white collar financial fraud occurs all the time and plenty of lawyers stretch their profession to (or beyond) the ethical breaking point — with devastating impacts on real people.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#86

Post by epistasis »

moonchrome wrote: post #52
ethanbond wrote: post #41

Earlier quoted context omitted.

Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve c…

> Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? If you can't control parameters so that the results you are claiming are reproducible you are just publishing noise ?

How would you know if parameters are controlled until other people try the same thing with other parameters?

Should a journal paper never be published until the field has established all the fundamental properties? How could it discover those fundamental properties without the publications?

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#87

Post by elliekelly »

mountainb wrote: post #60
willis936 wrote: post #8

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits…

In other fields where there is a chance for fraud, the honor system is not used. In academia, the attitude is just 'the honor system is good enough, WINK' as it relates to data. In other fields where there are opportunities for embezzlement or other abuses of trust everything is monitored, only some employees are authorized to do certain transactions, and many of those employees are bound by law and oath to not do th…

I think the damage is much more than just the integrity of the knowledge base. There are people who spend years focusing their (very expensive!) academic pursuits in reliance on these studies. They spend years of their lives researching and trying to replicate in vain. The damage to our collective knowledge is costly and unacceptable but there are also real people who are real victims of this fraud. It’s not just some theoretical cost borne by society at large.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#88

Post by linuxftw »

jbullock35 wrote: post #67

An important finding is buried in the linked blog post. It turns out that Ariely was hyping the fraudulent findings on NPR in February 2020. At the time, perhaps he didn't know that they were fraudulent. But he -did- know that there was great reason to doubt them. We can be confident of this point because, more than half a year earlier, he had submitted for publication a new article [1] that (a) failed to replicate t…

Let's not give NPR a pass here. They routinely present junk science as fact and never suffer any repercussions.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#90

Post by throwawaygh »

linuxftw wrote: post #88
jbullock35 wrote: post #67

An important finding is buried in the linked blog post. It turns out that Ariely was hyping the fraudulent findings on NPR in February 2020. At the time, perhaps he didn't know that they were fraudulent. But he -did- know that there was great reason to doubt them. We can be confident of this point because, more than half a year earlier, he had submitted for publication a new article [1] that (a) failed to replicate t…

Let's not give NPR a pass here. They routinely present junk science as fact and never suffer any repercussions.

As a general heuristic, social sciences research that tells a nice narrative and gets hyped to hell in TED talks/pop sci books/etc. is at best bullshit.
Post reply on HN