Noted study in psychology fails to replicate, crumbles with evidence of fraud • Hacker News Board

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#71

Post by sokoloff » Sun, Aug 22, 2021, 12:45 PM UTC

kortilla wrote: post #66

dalbasal wrote: post #46

Earlier quoted context omitted.

Let me continue the example... We found that students participating in the " 2hr active recess program " have meaningfully better health outcomes, psychological health outcomes, and better grades. It was tested faithfully in 25 schools over 10 years. It expands to other places. Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are alread…

> Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities. Elucidating confounding variables and controlling for them is the entire fucking job of a scientist. To your example, if they publish…

I suspect it’s the journalism step that introduces the error you’re objecting to. Proper scientists* will report their findings and methods. Journalists will tend to write the “Doing X leads to Y in humans” headlines.

Many times the first or first few teams won’t know what the confounding variables are and subsequent research is needed to find them. They should still report their initial “hmm, that’s novel” findings even if it later turns out that it’s explained by confounding variables.

*-No true Scotsman

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#72

Post by bawana » Sun, Aug 22, 2021, 12:48 PM UTC

Fake news in academia. They need to do it because their click bait does not drive ad revenue, rather it drives the awarding of grant money. Does the 'educational institution' where the researcher works then have to pay back the grant money? After all, everything in the lab is actually owned by the university and the grant is administered by the university and the university takes a hefty 'management fee'

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#73

Post by FabHK » Sun, Aug 22, 2021, 1:02 PM UTC

jononor wrote: post #27

atoav wrote: post #24

Earlier quoted context omitted.

How about instead of applying to journals with a finished study, people would apply with a proposal of a study and then publish the results independent of how spectacular/boring they turn out? Journals would then have to select studies based on their design and based on the question and scientists could just do science.

This is a good idea, and is known as "preregistration". https://en.wikipedia.org/wiki/Preregistration_(science) Hopefully it will become more common practice.

Well, it is preregistration plus a commitment by a journal to accept and publish the resulting paper (no matter the outcome of the study). That apparently is "Registered reports" (see same Wikipedia article), an idea I hadn't heard before.

Edited to add (from Wikipedia):

> Over 200 journals offer a registered reports option (Centre for Open Science, 2019),[33] and the number of journals that are adopting registered reports is approximately doubling each year (Chambers et al., 2019).[34]

> Nature Human Behaviour has adopted the registered report format, as it “shift[s] the emphasis from the results of research to the questions that guide the research and the methods used to answer them”.[36]

> European Journal of Personality defines this format: “In a registered report, authors create a study proposal that includes theoretical and empirical background, research questions/hypotheses, and pilot data (if available). Upon submission, this proposal will then be reviewed prior to data collection, and if accepted, the paper resulting from this peer-reviewed procedure will be published, regardless of the study outcomes.”[37]

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#74

Post by nyolfen » Sun, Aug 22, 2021, 1:05 PM UTC

dalbasal wrote: post #36

profstasiak wrote: post #14

I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility cri…

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the…

the replication crisis came to a head a few years ago when it was revealed that two thirds of social psychology findings failed to replicate -- this is not just empiricism taken too far, this is the mass production of false knowledge due to poor training at best or incentive misalignment at worst

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#75

Post by revolvingocelot » Sun, Aug 22, 2021, 1:09 PM UTC

Hmm? Academic fakery to support some financial unreality isn't exactly new. Remember how some economists figured they wouldn't include important chunks of the first world in an Excel calculation, in order to justify the grinding austerity favoured by right-wingers in the early 2010s? [0] They kept the data, they just decided not to include it in the calculations. Amazingly, actually including the data shows that the effects of a high debt-to-GDP on growth are negligible at best, rather than overwhelmingly bad.

Did they get anything more than a slap on the wrist? Are they still selling books based on the concept?

The breathless commentary regarding how perfidious psychology is threatens to sprain the muscles that govern the rolling of my eyes.

[0] https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#76

Post by subroutine » Sun, Aug 22, 2021, 1:11 PM UTC

observer20 wrote: post #61

Psychology like psychiatry is very similar to pseudoscience and I mean identical to the snake oil salesmen. Both of the fields attempt to “help” the unfortunate, who truly need help and placebos can arguably help "overcome" life factors. I was naive young from wishful thinking/optimism and thus fell prey to the mentality of the fields being positive for society; nowadays I realize the opposite. The sad thing is that…

Your assessment of psychology is flawed in several ways. The main concern of psychological research isn't to help someone. It's to understand how/why organisms think and behave under certain conditions. Psychological research can be conducted using the scientific method like any other science field. To say "Psychology", as a whole, is similar to snake oil, ignores thousands of perfectly sound findings. Fraud is certainly not unique to the field of psychology. Nor are the pressures to publish positive findings.

Your final thoughts about 'life outcomes from genetics' is incoherent.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#77

Post by auganov » Sun, Aug 22, 2021, 1:13 PM UTC

throwthere wrote: post #51

auganov wrote: post #34

If top scientists are willing to outright manufacture data, how many would be willing to intentionally follow unsound logic? As in dishonestly claiming some (legitimate) data implies something that it doesn't or intentionally avoiding alternative explanations. You'd have to assume there's even more of that and all other kinds of soft fraud going on. Ideally planning data production, execution of data production and a…

These are top social scientists and, unfortunately, their field is a total dumpster fire at the moment. See https://www.nature.com/articles/s41562-018-0399-z

If you're implying the replication crisis is limited to social sciences, that's too narrow. You'd expect to see these problems in any field that primarily relies on hard to gather data that is far removed from the underlying phenomenon. Many sciences that aren't strictly "soft" have "soft" fields. Say, Biology->drug research.

Also not to argue semantics but when you hear the word "scientist" in common parlance it almost exclusively refers to people doing that kind of research. The "sciences" that don't have these problems tend to not be too big on using the word "science".

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#78

Post by dalbasal » Sun, Aug 22, 2021, 1:25 PM UTC

kortilla wrote: post #66

dalbasal wrote: post #46

Earlier quoted context omitted.

Let me continue the example... We found that students participating in the " 2hr active recess program " have meaningfully better health outcomes, psychological health outcomes, and better grades. It was tested faithfully in 25 schools over 10 years. It expands to other places. Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are alread…

> Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities. Elucidating confounding variables and controlling for them is the entire fucking job of a scientist. To your example, if they publish…

>> Elucidating confounding variables and controlling for them is the entire fucking job of a scientist.

No. This is methodology. The job of a scientist is to advance scientific knowledge. This is as vague as it needs to be.

Popper didn't do science. He did philosophy. He also did a fine job of formalising methods, and abstracting the methods scientists were using into a Scientific Method. This scientific method is a great achievement and a wonderful, multipurpose probe. That does not mean it fits in all orifices. The job of a scientist is not to continue hammering until the probe fits.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#79

Post by analog31 » Sun, Aug 22, 2021, 1:31 PM UTC

dalbasal wrote: post #46

rwbhn wrote: post #39

Earlier quoted context omitted.

If the results are not replicable, how can they be actionable?

Let me continue the example... We found that students participating in the " 2hr active recess program " have meaningfully better health outcomes, psychological health outcomes, and better grades. It was tested faithfully in 25 schools over 10 years. It expands to other places. Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are alread…

>>>> Yet, we as humans do manage to make sense of the world. We run schools, companies. We learn more about how to do those things. Formal studies are an important part of learning about how to do those things. Scientific mindset and methodologies are useful to this. It's a lot better than just philosophizing the whole thing. The type of knowledge we gain, OTOH, isn't like F=MA. It never will be.

I think it has to to be held out as a possibility that there is actually no science behind how we run schools and companies. One way this could happen is if non-scientific ideas simply overwhelm the ones that are grounded in good science.

The fact that it isn't like F=MA also means it will never advance beyond producing collections of amusing factoids that are unrelated to one another and potentially false.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#80

Post by CrazyCatDog » Sun, Aug 22, 2021, 1:33 PM UTC

As a business academic, I can absolutely confirm that popular press attention corrupts us just as much as power corrupts political leaders. I can’t speak outside of b-schools, but suspect it’s no different.

If anyone is privy to a platforms that would allow me to share proprietary data such that skeptics can run aggregated statistical analyses whilst limiting the ability to surface results with insufficient samples (to prevent scrapping the the proprietary data—if not proprietary then there is no excuse for not fully sharing) please comment!

If not, YC hopefuls here is your chance to ACTUALLY make the world a better place!!! Happy to use my platform to recruit as many social scientists as you need to secure funding for a POC—seriously, ping me!

Must haves:

* Easily palatable settings for data obfuscation (non-tech corporate GC must be able to grok the level of obfuscation)

* Trivial sliders to change level of obfuscation (eg min number of observation to run a “test”)

* Messaging system to facilitate requests and permission granting between authors and skeptics

* Enterprise-level data retention and documentation, such that changes to the “official” data set area reflected publicly

* Cloud based R, Python, Stata functionality (else the solution would have to run locally in such a way that the granular data cannot be accessed directly on local machine—and this would likely be a non-starter for corporate GC’s). Ideally, one could reuse their local license for proprietary software (eg Stata), if not then said vendors will delight in an additional “attach” license to charge researchers even more (see below, there are beacoup bucks that will gladly be spent on this)

Business model: researcher’s academic institution would (gladly) pay (if you get pushback, call the university’s academic HR department, find out who is in charge (Ie associate provost of research or HR) and email them asking they are prepared we’re to shine a spotlight on their own “Andrew Bird and Stephen Karolyi” like scholars). They will call you in 5 min.

GitHub will acquire you post-POC upon enlisting 2-3 major universities (again, I know what higher-Ed procurement cycle is like, contracts will be signed record time). If GitHub fails, Google scholar will acquire you.