Psychology like psychiatry is very similar to pseudoscience and I mean identical to the snake oil salesmen. Both of the fields attempt to “help” the unfortunate, who truly need help and placebos can arguably help "overcome" life factors. I was naive young from wishful thinking/optimism and thus fell prey to the mentality of the fields being positive for society; nowadays I realize the opposite. The sad thing is that society has morally suffered tremendously from both fields' actions & thoughts throughout history. Persons having poor life outcomes from genetics, where their time is squandered by misguided trust from themselves or loved ones and where what they need isn't there while sometimes they have a lot worse because of the fields.
Noted study in psychology fails to replicate, crumbles with evidence of fraud
61–70 of 110 posts
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#62Earlier quoted context omitted.
If the results are not replicable, how can they be actionable?
Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve c…
Or it means there was fraud, which is why they are ultimately not able to be separated.
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#63I have several books of Ariely. I enjoy reading psychology, but I stopped reading pop psychology books and go for academic books or textbooks or individual articles. The amount of times I read about spectacular counter-intuitive study that "changes everything" and later find out it was fraudulent or failed to replicate makes me really doubt all of my knowledge I have from reading about psychology. Reproducibility cri…
It is wrong to believe just because a result cannot be replicated the original result is wrong or fraudulent. It can be totally normal and is to be expected that a replication study fails to yield the same result just out of chance etc. Sometimes things are even more complicated than you already believe them to be. And it may take more than just a replication study to say a study is wrong or fraudulent.
It is wrong if it doesn’t replicate; it might not be fraudulent. But it is not right.
If it doesn’t replicate, it should be relegated to “less than observational” level evidence. Why would you believe that the assumed intervention had the claimed effect, if it doesn’t have it when you try again?
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#64Psychology like psychiatry is very similar to pseudoscience and I mean identical to the snake oil salesmen. Both of the fields attempt to “help” the unfortunate, who truly need help and placebos can arguably help "overcome" life factors. I was naive young from wishful thinking/optimism and thus fell prey to the mentality of the fields being positive for society; nowadays I realize the opposite. The sad thing is that…
I think this is over-sold, but psychology as a field is absolutely closer to philosophy with a veneer of science put on top than the field would have you believe from the outside, and psychology as a job is absolutely not science based day-to-day.
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#65You don't even need the graphs, miles driven per year is a sum. Sums converge to gaussians yet here the deviation was half of mean which means they used a uniform variable generator for miles.
> yet here the deviation was half of mean which means they used a uniform variable What am I missing? The mean of a normal distribution can be any value. A normal with mean 2 and deviation 1 has deviation equal to half of the mean. Is this related to the fact that miles driven can't be negative (unlike a sample from a genuine normal)?
I think this is an overzealous statement of the central limit theorem. A log-normal distribution would probably be a better model in this case.
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#66Earlier quoted context omitted.
If the results are not replicable, how can they be actionable?
Let me continue the example... We found that students participating in the " 2hr active recess program " have meaningfully better health outcomes, psychological health outcomes, and better grades. It was tested faithfully in 25 schools over 10 years. It expands to other places. Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are alread…
> Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities.
Elucidating confounding variables and controlling for them is the entire fucking job of a scientist.
To your example, if they publish a paper that explicitly highlights that it worked well for sedentary kids with typical diets of X, Y, and Z in region Q, then it failing to be a found in other environments isn’t a problem.
It’s a problem when these clowns do a test on one specific group of 20 people and then generalize to, “doing X leads to Y in humans”.
The answer is absolutely not loosening up the constraints. It’s to keep lambasting people who generalize before anything has been reproduced widely.
Popper’s science is fine. Morons who don’t realize what they measured are the problem.
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#67An important finding is buried in the linked blog post. It turns out that Ariely was hyping the fraudulent findings on NPR in February 2020. At the time, perhaps he didn't know that they were fraudulent. But he -did- know that there was great reason to doubt them. We can be confident of this point because, more than half a year earlier, he had submitted for publication a new article [1] that (a) failed to replicate the original findings, and (b) uncovered a massive problem with baseline measurements in the original study, suggesting that the randomization in that study might never even have taken place. Indeed, the whole point of the new article was to make these critical points about the older article!
Even if Ariely had no role in the creation of the fake data, it seems hard to defend his behavior on NPR: hyping a study that he knew to be extremely problematic.
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#68The 2012 article states:
> Specifically, we ran analyses examining whether the two conditions differed in the number of instances wherein reported odometer mileages ended with 0, 5, 00, 50, 000, or 500. Numbers that end with these digits indicate a higher likelihood that customers simply estimated their mileage. We detected no statistically significant differences between our two conditions in the instances in which these endings appeared (pooled measure: treatment, 19.9% vs. control, 20.8%; χ2 = 2.5, P = 0.12).
1. What does this mean? (“Pooled measure”) Wouldn't a mileage ending in 500 also end in 0 and 00?
2. If the numbers are uniformly randomly generated, shouldn’t the odds of ending in 0 be 10%, not 20%? Or is this an equal mix of fraudulent and real data in both arms?
Just curious because this is the one of the only bits of data for the fraudulent study in the 2012 article.
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#69What if humans were generally so unique that there were no useful generalizations in psychology. (E.g every attempt at an experiment like this was just noise)
How would that affect the field?
Re: Noted study in psychology fails to replicate, crumbles with evidence of fraud
#70Earlier quoted context omitted.
Incentivizing publishing negative results is a challenge.
There needs to be a disincentive to not publishing. The journal that publishes failed science would be the biggest and most important scientific journal IMO.
There's a few negative results journals now - https://www.negative-results.org/ for example.
The only concern I have around a disincentive to not publishing is that in some fields, the peer review process can be quite flawed and effectively hold back worthwhile publications for various reasons - sometimes there's too much of an expectation of specific ingrained methodologies, or an expectation of certain "doctrines" to be adhered to.
There can also be some significant gaps between disciplines, where neither discipline wants to publish the work, as they both feel it belongs in the other discipline, which effectively leaves only the more generic (and usually less prestigious) "any inter-disciplinary research" journals.
That becomes a problem if institutions are focused on publication metrics and venues as a priority, as it means the institutional incentives drive people towards or away from these kinds of work. Ideally we need negative results published in the same venues as positive results (in my view), so those doing this valuable work don't get "relegated" to the negative results journals, which will no doubt get less attention at tenure review time, or from grant funders looking at publications etc.