Noted study in psychology fails to replicate, crumbles with evidence of fraud • Hacker News Board

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#91

Post by dalbasal » Sun, Aug 22, 2021, 2:04 PM UTC

rossdavidh wrote: post #83

dalbasal wrote: post #36

Earlier quoted context omitted.

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the…

I think the current situation calls for a more rigorous application of Popperian standards, not less. Nothing about "this study applies to this specific situation" is counter to Popper. However, if it fails to replicate in other situations, it tends to suggest at the very least "might be too specific to this situation to have practical application". This happens in software as well. If my system works on my specific…

>> Nothing about "this study applies to this specific situation" is counter to Popper. However, if it fails to replicate in other situations, it tends to suggest at the very least "might be too specific to this situation to have practical application".

Maybe not technically, but it does imply that "this specific situation" should be defined specifically, hence controls. I agree that broad applicability, and replicability are strong indicators of truth value. That said, there are pursuits of rational knowledge outside of Popperian ideals. That's not represented in the way Science, as an institutions works.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#92

Post by regardsop » Sun, Aug 22, 2021, 2:07 PM UTC

tux3 wrote: post #55

smitty1e wrote: post #33

Earlier quoted context omitted.

Incentivizing publishing negative results is a challenge.

This has potential consequences I don't like. Once negative results are incentivized the same way as positive results, cheaters could have a perverse incentive to post fraudulent failures to replicate, too. I think I agree that we need much more transparency and scientific rigor, first. Post the data whenever possible, incentivize people for creating good study protocols that make it harder for any single person to f…

"If negative results are published like other information, people could have a incentive to replicate post fraudulent failures"?

Um sounds worse, (OT:) to add something a simpified (strict stripped) example -hope practicable:

You work at a company. Orders are comming in, wares are placed as 'reserved' till paid, than shipped. With a worker perspective, you check the emails with orders ('the customer whishes'), sending 'bills' and the wares if, or if not been paid - 'you've been paid'. Viewed by the company-manager your position may a) be valid oder come b) 'overvalued' (...)

Stucked in this example, you talked to other 'employeees' during break and 'there was a time meeting on weekends', so you talked about work and there was a (special) kind of 'more transparency' than without peer review. ^^

Regards, hope this helps... (-:

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#93

Post by anchpop » Sun, Aug 22, 2021, 2:10 PM UTC

CrazyCatDog wrote: post #80

As a business academic, I can absolutely confirm that popular press attention corrupts us just as much as power corrupts political leaders. I can’t speak outside of b-schools, but suspect it’s no different. If anyone is privy to a platforms that would allow me to share proprietary data such that skeptics can run aggregated statistical analyses whilst limiting the ability to surface results with insufficient samples (…

Hmm, I wonder if one could use differential privacy to make mathematically certain that no individual data point could leak. Although that may not be a strong enough guarantee

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#94

Post by ethanbond » Sun, Aug 22, 2021, 2:11 PM UTC

DanielBMarkham wrote: post #44

ethanbond wrote: post #41

Earlier quoted context omitted.

Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve c…

Yes. Find it. And when you do, you'll have something reproducible. The purpose of science is to advance the species. Given this, when that happens, this other thing(s) occur (or occur with a certain frequency) I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had gene…

To be clear, in this case there appears to have been fraud ie actual falsification of data. That’s very different from the broader replication crisis, which is more of the shape of unknown parameters. The two get blurry in some places due to eg publication incentives pushing people’s interpretation of legitimate data into semi-fraudulence. But there are still two ends (at least) on the spectrum of non-replicable science.

I think framing it in this lens is helpful because it points toward a solution: capturing more of the parameters more explicitly and varying them more deliberately. If we just say, “oh x y z studies aren’t replicable, field [x y z] is bullshit,” we are ending our inquiry prematurely. We are ending it at the falsification of Newton’s initial theory and saying that because leaves fall at a different rate, that gravity doesn’t exist.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#95

Post by anchpop » Sun, Aug 22, 2021, 2:30 PM UTC

ghyiihdruh wrote: post #20

What's the purpose of this? It will just give ammo to anti-science Trump like people. It's harmful. This kind of stuff should be handled as quietly as possible, and definitely not in public. Science is already under attack, we need not undermine it from the inside too.

I mean, if anti-science trump people use this to be skeptical of results in psychology and sociology, how wrong would they be? these fields are genuinely corrupt and generate unreliable results. We already know the solution: preregistration and registered reports. That may not actually be enough because there are other issues (e.g. almost all studies that attempt to “control for x” or similar are subtly flawed and should be replaced with proper RCTs), but it would at least be a huge and obvious and inexpensive step in the right direction, and the fact that it hasn’t been implemented yet shows that the field is immature and broadly shouldn’t be trusted.

(compare to medical research, which has embraced prereplication because the FDA is actually serious about safety and efficacy and had the good sense to require it. but the FDA is anchored in the real world - if they get it wrong people may die and it’s on their hands. If psychologists get a result wrong, they may generate a bombastic paper for themselves, do a ted talk, go speak on NPR, then write a popsci book, and generally boost their career. So the incentive is for the FDA to be right and for the psychologist to be wrong, so it’s not surprising that that’s what we see.)

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#96

Post by myWindoonn » Sun, Aug 22, 2021, 2:37 PM UTC

auganov wrote: post #77

throwthere wrote: post #51

Earlier quoted context omitted.

These are top social scientists and, unfortunately, their field is a total dumpster fire at the moment. See https://www.nature.com/articles/s41562-018-0399-z

If you're implying the replication crisis is limited to social sciences, that's too narrow. You'd expect to see these problems in any field that primarily relies on hard to gather data that is far removed from the underlying phenomenon. Many sciences that aren't strictly "soft" have "soft" fields. Say, Biology->drug research. Also not to argue semantics but when you hear the word "scientist" in common parlance it alm…

Cosmology is the first field which comes to mind; it fits your criteria, but is far harder (as a science) than psychology and far more reliable. Psychologists can't define "mind" and change their entire profession every few decades, but "sun" has not changed much as a phenomenon for a long time.

The reason why the social sciences (social psychology, criminology, economics, advertising/marketing) are hit hardest by the replication crisis is because they were not actually sciences! They are ways for our society to police its members' behavior (relationships, crime, money, attention) and they pretend to be science in order to gain prestige.

There's also, finally, the matter of scale and relevance. When a chemist reports that a rare chemical reaction doesn't perform in the lab as expected, then this is a failed replication; however, chemistry labs regularly replicate entire textbooks' worth of material for entire classes of undergraduates. Meanwhile, some of the most important claims in psychology, like priming, are failing to replicate.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#97

Post by myWindoonn » Sun, Aug 22, 2021, 2:42 PM UTC

observer20 wrote: post #61

Psychology like psychiatry is very similar to pseudoscience and I mean identical to the snake oil salesmen. Both of the fields attempt to “help” the unfortunate, who truly need help and placebos can arguably help "overcome" life factors. I was naive young from wishful thinking/optimism and thus fell prey to the mentality of the fields being positive for society; nowadays I realize the opposite. The sad thing is that…

It's worse than that; Pirsig reminds us that psychiatrists are literally thought police. From this perspective, psychology is a mechanism for justifying psychiatric abuses. After all, if psychology can show that minds exist, that there are normal and abnormal minds, that there are correct and incorrect modes of thought, then psychiatry has only to show that certain incorrect modes of thought form clustered disorders and syndromes. The resulting system can punish people for expressing anything which society doesn't like, from sexuality to politics to a desire not to be overworked.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#98

Post by keutoi » Sun, Aug 22, 2021, 2:45 PM UTC

dalbasal wrote: post #78

kortilla wrote: post #66

Earlier quoted context omitted.

> Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities. Elucidating confounding variables and controlling for them is the entire fucking job of a scientist. To your example, if they publish…

>> Elucidating confounding variables and controlling for them is the entire fucking job of a scientist. No. This is methodology. The job of a scientist is to advance scientific knowledge. This is as vague as it needs to be. Popper didn't do science. He did philosophy. He also did a fine job of formalising methods, and abstracting the methods scientists were using into a Scientific Method. This scientific method is a…

How do you advance scientific knowledge without using a Scientific Method?

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#99

Post by dalbasal » Sun, Aug 22, 2021, 2:58 PM UTC

keutoi wrote: post #98

dalbasal wrote: post #78

Earlier quoted context omitted.

>> Elucidating confounding variables and controlling for them is the entire fucking job of a scientist. No. This is methodology. The job of a scientist is to advance scientific knowledge. This is as vague as it needs to be. Popper didn't do science. He did philosophy. He also did a fine job of formalising methods, and abstracting the methods scientists were using into a Scientific Method. This scientific method is a…

How do you advance scientific knowledge without using a Scientific Method?

I suppose any method used to advance scientific knowledge without is a scientific method, by definition. That's besides the point though.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#100

Post by tpoacher » Sun, Aug 22, 2021, 2:58 PM UTC

nyolfen wrote: post #74

dalbasal wrote: post #36

Earlier quoted context omitted.

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the…

the replication crisis came to a head a few years ago when it was revealed that two thirds of social psychology findings failed to replicate -- this is not just empiricism taken too far, this is the mass production of false knowledge due to poor training at best or incentive misalignment at worst

It is not even about malice. It is a straightforward consequence of how hypothesis testing is conducted. Greenland et al have a nice section on this:

> Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % power against a key alternative when using a 0.05 significance level, although in execution often have less power due to unanticipated problems such as low subject recruitment. Thus, if the alternative is correct and the actual power of two studies is 80 %, the chance that the studies will both show P ≤ 0.05 will at best be only 0.80(0.80) = 64 %; furthermore, the chance that one study shows P ≤ 0.05 and the other does not (and thus will be misinterpreted as showing conflicting results) is 2(0.80)0.20 = 32 % or about 1 chance in 3. Similar calculations taking account of typical problems suggest that one could anticipate a ‘‘replication crisis’’ even if there were no publication or reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs of ‘‘significant’’/‘‘nonsignificant’’ or ‘‘reject’’/‘‘accept.’’