Noted study in psychology fails to replicate, crumbles with evidence of fraud • Hacker News Board

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#101

Post by mensetmanusman » Sun, Aug 22, 2021, 3:03 PM UTC

sennight wrote: post #84

mensetmanusman wrote: post #69

What if humans were generally so unique that there were no useful generalizations in psychology. (E.g every attempt at an experiment like this was just noise) How would that affect the field?

It would probably focus on measuring the degree to which individuals differ, assigning explanatory variables, and figuring out the theoretical maximum (as brain chemistry sets a pretty hard limit). But that doesn't really matter, because we know that people can very easily be generalized - there is an entire industry devoted to it: advertising. Many years ago I had to design a process to pre-stage inventory based on…

Makes sense for consumer behavior since there are a finite set of purchase choices. Things like ethics and behavior choices are fuzzy though…

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#102

Post by tux3 » Sun, Aug 22, 2021, 3:35 PM UTC

regardsop wrote: post #92

tux3 wrote: post #55

Earlier quoted context omitted.

This has potential consequences I don't like. Once negative results are incentivized the same way as positive results, cheaters could have a perverse incentive to post fraudulent failures to replicate, too. I think I agree that we need much more transparency and scientific rigor, first. Post the data whenever possible, incentivize people for creating good study protocols that make it harder for any single person to f…

"If negative results are published like other information, people could have a incentive to replicate post fraudulent failures"? Um sounds worse, (OT:) to add something a simpified (strict stripped) example -hope practicable: You work at a company. Orders are comming in, wares are placed as 'reserved' till paid, than shipped. With a worker perspective, you check the emails with orders ('the customer whishes'), sendin…

I'm not sure I understand your point, but if you're saying that posting negative results is good, then yes, I agree!

I'm only trying to make a point about the order in which we should shift incentives vs increase fraud-resistance requirements for experimental protocols.

I think the latter is more important, and should be done first.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#103

Post by sennight » Sun, Aug 22, 2021, 3:49 PM UTC

mensetmanusman wrote: post #101

sennight wrote: post #84

Earlier quoted context omitted.

It would probably focus on measuring the degree to which individuals differ, assigning explanatory variables, and figuring out the theoretical maximum (as brain chemistry sets a pretty hard limit). But that doesn't really matter, because we know that people can very easily be generalized - there is an entire industry devoted to it: advertising. Many years ago I had to design a process to pre-stage inventory based on…

Makes sense for consumer behavior since there are a finite set of purchase choices. Things like ethics and behavior choices are fuzzy though…

> behavior choices

I really wish this weren't the case, but "choice" isn't a thing - as far as anyone can tell. All the complexity on display in the world appears to be a result of feedback loops, it really doesn't take much to generate a continuous low interval stream of pseudo random feedback - predictable after enough observation. People who regularly purchase chunky peanut butter and cat litter have a significantly higher probability of driving a VW bug. That isn't the result of limited alternatives.

https://en.wikipedia.org/wiki/Linear-feedback_shift_register

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#104

Post by observer20 » Sun, Aug 22, 2021, 4:45 PM UTC

subroutine wrote: post #76

observer20 wrote: post #61

Psychology like psychiatry is very similar to pseudoscience and I mean identical to the snake oil salesmen. Both of the fields attempt to “help” the unfortunate, who truly need help and placebos can arguably help "overcome" life factors. I was naive young from wishful thinking/optimism and thus fell prey to the mentality of the fields being positive for society; nowadays I realize the opposite. The sad thing is that…

Your assessment of psychology is flawed in several ways. The main concern of psychological research isn't to help someone. It's to understand how/why organisms think and behave under certain conditions. Psychological research can be conducted using the scientific method like any other science field. To say "Psychology", as a whole, is similar to snake oil, ignores thousands of perfectly sound findings. Fraud is certa…

No, the sentence "persons having 'poor life outcomes' from genetics" isn't incoherent but maybe to a simpleton it happens to be.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#105

Post by stewbrew » Sun, Aug 22, 2021, 4:59 PM UTC

beagle3 wrote: post #63

stewbrew wrote: post #43

Earlier quoted context omitted.

It is wrong to believe just because a result cannot be replicated the original result is wrong or fraudulent. It can be totally normal and is to be expected that a replication study fails to yield the same result just out of chance etc. Sometimes things are even more complicated than you already believe them to be. And it may take more than just a replication study to say a study is wrong or fraudulent.

It is wrong if it doesn’t replicate; it might not be fraudulent. But it is not right. If it doesn’t replicate, it should be relegated to “less than observational” level evidence. Why would you believe that the assumed intervention had the claimed effect, if it doesn’t have it when you try again?

In the present case, replication didn't fail because of statistics but because the experiment was impossible to reproduce.

If it fails to replicate certain statistical outcomes, things are more complicated.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#106

Post by CrazyCatDog » Sun, Aug 22, 2021, 6:09 PM UTC

anchpop wrote: post #93

CrazyCatDog wrote: post #80

As a business academic, I can absolutely confirm that popular press attention corrupts us just as much as power corrupts political leaders. I can’t speak outside of b-schools, but suspect it’s no different. If anyone is privy to a platforms that would allow me to share proprietary data such that skeptics can run aggregated statistical analyses whilst limiting the ability to surface results with insufficient samples (…

Hmm, I wonder if one could use differential privacy to make mathematically certain that no individual data point could leak. Although that may not be a strong enough guarantee

Yes, I’ve seen it on websites (eg GMAC) but the operators available are trivial (e.g. mean, median)… system will not process the request if the sample is too small (I think the minimum sample size is actually a function of how much you pay)

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#107

Post by goatlover » Sun, Aug 22, 2021, 8:41 PM UTC

dalbasal wrote: post #48

ethanbond wrote: post #41

Earlier quoted context omitted.

Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve c…

I disagree. You're not going to find the parameter which makes a high school nutrition program deterministic. It shouldn't be the goal.

Which means you can’t generalize to other high schools. If the scientific result doesn’t generalize, is it still science? How can we use the science to improve HS nutrition?

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#108

Post by goatlover » Sun, Aug 22, 2021, 8:44 PM UTC

stewbrew wrote: post #43

profstasiak wrote: post #14

I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility cri…

It is wrong to believe just because a result cannot be replicated the original result is wrong or fraudulent. It can be totally normal and is to be expected that a replication study fails to yield the same result just out of chance etc. Sometimes things are even more complicated than you already believe them to be. And it may take more than just a replication study to say a study is wrong or fraudulent.

It means the original result is an anomaly that doesn’t fit into a scientific understanding. Maybe there was an error in the experimental setup, or some unknown factor was at play, or the situation was unique. Whatever the case, nothing scientific can be determined if it can’t be replicated. It’s akin to the WOW SETI signal.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#109

Post by [deleted] » Sun, Aug 22, 2021, 9:09 PM UTC

tpoacher wrote: post #100

nyolfen wrote: post #74

Earlier quoted context omitted.

the replication crisis came to a head a few years ago when it was revealed that two thirds of social psychology findings failed to replicate -- this is not just empiricism taken too far, this is the mass production of false knowledge due to poor training at best or incentive misalignment at worst

It is not even about malice. It is a straightforward consequence of how hypothesis testing is conducted. Greenland et al have a nice section on this: > Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % powe…

[deleted]

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#110

Post by SNosTrAnDbLe » Mon, Aug 23, 2021, 4:00 AM UTC

I am wondering how much of such non replicated studies leaks into the decisions and recommendations by federal organizations like the CDC or ECDC or WHO. It creeps me out to think that some organizations adopted some of the ideas outlined in this person's book.