Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Noted study in psychology fails to replicate, crumbles with evidence of fraud

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

51–60 of 110 posts

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#51

Post by throwthere »

auganov wrote: post #34

If top scientists are willing to outright manufacture data, how many would be willing to intentionally follow unsound logic? As in dishonestly claiming some (legitimate) data implies something that it doesn't or intentionally avoiding alternative explanations. You'd have to assume there's even more of that and all other kinds of soft fraud going on. Ideally planning data production, execution of data production and a…

These are top social scientists and, unfortunately, their field is a total dumpster fire at the moment. See https://www.nature.com/articles/s41562-018-0399-z

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#52

Post by moonchrome »

ethanbond wrote: post #41
rwbhn wrote: post #39

Earlier quoted context omitted.

If the results are not replicable, how can they be actionable?

Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve c…

> Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to?

If you can't control parameters so that the results you are claiming are reproducible you are just publishing noise ?

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#53

Post by ImaCake »

throwthere wrote: post #42
knuthsat wrote: post #10

You don't even need the graphs, miles driven per year is a sum. Sums converge to gaussians yet here the deviation was half of mean which means they used a uniform variable generator for miles.

Interesting point. So the original paper (2012) didn't give a table or anything, but said: "Customers who signed at the beginning on average revealed higher use (M = 26,098.4, SD = 12,253.4) than those who signed at the end [M = 23,670.6, SD = 12,621.4; F(1, 13,485) = 128.63, P Was M = 26,098.4, SD = 12,253.4 enough to infer a uniform distribution?

> Was M = 26,098.4, SD = 12,253.4 enough to infer a uniform distribution?

No unfortunately it is not. It's perfectly plausible to have a normal distribution with mean = 26k and sd = 12k. Although those numbers do look kinda weird, but nothing you could verify. I am not sure what the F(*) means here, maybe an F statistic? But that seems wrong if you are comparing two normally distributed samples, you might expect a T statistic.

To verify the distribution you would need a histogram or you could get fancy with a qqplot. You could also try a statistical test for a normal distribution but these fail on large sample sizes so the visual plots are your best bet.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#54

Post by gus_massa »

zhdc1 wrote: post #2

I'm pretty sure this was discussed in an earlier HN post. However, I want to point out that "not replicating" doesn't necessarily equate to bad science (or, more accurately, bad practice on the part of the principal investigators irt accepted best practices in their field). There are a bunch of studies in the social sciences that failed to replicate simply because a p value was slightly below a threshold target, even…

> because a p value was slightly below a threshold target

If some area is getting too many "statistically significant" flukes, they must choose a lower threshold. For example in particle physics they use 5 sigmas that is an insane low threshold, because otherwise they would have to announce a fake particle every month and raise a retraction a few month later.

> even though the coefficient direction/magnitude &c... agreed with the earlier study

This is a problem due to report and publication bias. Last month there was a discussion abut the relation of lead and crime https://news.ycombinator.com/item?id=28016921 . Everybody knows that lead is bad. In that meta analysis, most studies show a very low effect, and only a few have a strong effect. The problem is that a possible explanations is that flukes that show a strong bad effect are published, but flukes that show that plumber is good for people are silently discarded.

Another field with a lot of report and publication bias are drugs against covid-19. For example look at the graph near the bottom of https://news.ycombinator.com/item?id=27852130 How many of these studies are not even statically significant?

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#55

Post by tux3 »

smitty1e wrote: post #33
newsclues wrote: post #13

Earlier quoted context omitted.

Publishing negative results is important to solving this issue.

Incentivizing publishing negative results is a challenge.

This has potential consequences I don't like. Once negative results are incentivized the same way as positive results, cheaters could have a perverse incentive to post fraudulent failures to replicate, too.

I think I agree that we need much more transparency and scientific rigor, first. Post the data whenever possible, incentivize people for creating good study protocols that make it harder for any single person to fake the data without it being obvious. Once we have enough confidence in that, let's do preregistration, sure. And then negative results should be as incentivized as positive ones, absolutely.

I worry about the order in which we want to do these things. If people are faking data, we need to make this specific step harder to hide from reviewers.

If you incentivize people to fail to replicate each other without solving the lag between fraudulent data and retractation, you will get fraudulent negative results.

That could cause a lot of confusion. If the replication crisis itself starts to be fraudulent, scientific credibility is further damaged, negative results will be less credible, and fraudsters will profit from the confusion.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#56

Post by newsclues »

smitty1e wrote: post #33
newsclues wrote: post #13

Earlier quoted context omitted.

Publishing negative results is important to solving this issue.

Incentivizing publishing negative results is a challenge.

There needs to be a disincentive to not publishing.

The journal that publishes failed science would be the biggest and most important scientific journal IMO.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#57

Post by ImaCake »

elevenoh wrote: post #11

I love polymarket. Prediction markets on topics like this are about the only way to discern truth from afar.

Wouldn't polymarket be subject to incomplete information? Thus leading to bias. Polymarket would just reflect the market sentiment given a fairly robust analysis of the available information, admittedly a pretty useful idea.

The other issue is that a very niche bet/topic might only have a small sample of analysts placing bets. This would mean you are subject to increased risk of bias from a single bad analyst. Or simply large variance due to small sample size.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#58

Post by dalbasal »

jfengel wrote: post #49
ethanbond wrote: post #41

Earlier quoted context omitted.

Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve c…

Generalizing further, it touches on the Demarcation Problem. What is the difference between scientific hypothesis revision and pseudoscientific goalpost moving? Surely there is one, but it's maddeningly hard to pin down. Especially when the experiments are hard to perform and control properly, which applies both to psychology and astronomy.

This kind of demarcation usually hints that you are dealing with a subject that is not as concrete as you had supposed. "Science" doesn't really have a concrete definition, a pseudoscience just means stuff that is pretending to be science.

Remember that scientists existed before philosophy of science, including formalisations of The Scientific Method. Darwin wasn't following the scientific method, the method followed him.

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#59

Post by beagle3 »

dalbasal wrote: post #48
ethanbond wrote: post #41

Earlier quoted context omitted.

Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve c…

I disagree. You're not going to find the parameter which makes a high school nutrition program deterministic. It shouldn't be the goal.

That’s a strawman. Reproducibility does not require determinism.

If you can’t find the parameters that make it reproducible, what value does it have in the first place?

Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud

#60

Post by mountainb »

willis936 wrote: post #8

In private I have heard of science done that is suspected to be fraudulent. The results are so intriguing and promising as to send multiple grad students across multiple groups barreling into replicating and expanding the results. What happened? Years of time wasted with no publications. PIs and journals aren't interested in publishing negative results. The person who lied in their research is reaping career benefits…

In other fields where there is a chance for fraud, the honor system is not used. In academia, the attitude is just 'the honor system is good enough, WINK' as it relates to data. In other fields where there are opportunities for embezzlement or other abuses of trust everything is monitored, only some employees are authorized to do certain transactions, and many of those employees are bound by law and oath to not do things like embezzle or falsify records.

This doesn't always work but having the capacity to disbar lawyers for, say, embezzling trust accounts or remove a CPA's license and put them in prison for falsifying records, it certainly has a deterrent effect on similar crimes of trust.

How much worse would white collar crime be if everything was on the 'honor system' and there was no prison time for stealing from customers/clients/shareholders? That's what we have in academia: crime won, so now syndicates of crooks run the system. With other abuses of trust the material consequences are more readily apparent: someone's bank account is empty that should not be. With this abuse of trust, the damage is more to the integrity of the purportedly rational knowledge base. It's more of a 'reign of error' than a reign of terror.

Post reply on HN