Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
1–10 of 102 posts
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#2I want a chart like that but for journalism.
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#3I'm cynical and have no trouble believing that results were twisted immensely prior to the rule change in 2000. The incentives are huge: profit, prestige, or simply job security.
I do wonder if this chart misrepresents something, though: there are studies that produce incidental--but genuinely valuable--discoveries. It's unclear to me if that accounts for the pre-2000 results or not. With the new rules, would there have to be another study stating the new objective?
I'm not questioning that bad science occurs, but I am questioning what this graph really tells us.
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#4I want a chart like that but for journalism.
I don't follow this comment. What would you plot on the vertical axis?
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#5Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#6The downside of this though is that unexpected positive results sometimes get buried. For instance, clinical trials have to report all adverse effects but not positive effects. We've had drugs tested for symptom X fail to have any effect, while causing bald people to have their hair grow back. Yet the company we were testing it for wasn't interested and those results never made it into the public domain.
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#7Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.
Aren't they equivalent in their potential to allow post-hoc data mining?
Whether it's a clinical trial, or a psychological study with n participants, or a look at how changes in x soil conditions affects the y tree. Anything where data samples are taken and statistically analyzed is prone to p-hacking and after-the-fact hypothesis changes.
This example happens to be for a narrow set of studies: clinical trials for cardiovascular drugs/supplements. The take away from that graph applies broadly.
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#8I 100% agree with the headline; especially for research papers even more than clinical trials.
However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at.
But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials), science is somewhat additive. Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? With the advances in treatments for cardiovascular disease couldn't the problem just be harder?
And look at that "harm" study. Turns out it's one of many from the Women's Health Initiative [1], and it's the test of whether hormone therapy for post-menopausal women causes heart problems: that is, it's not a test of a drug to prevent heart conditions, but a test of side effects.
How many other of these studies are like that: studies about effects on the heart, not trials for new drugs to treat the heart? How did the ratio change before and after 2000?
In any case, don't trust every popular news article you read about science, particularly if it's written by Kevin Drum and posted on Mother Jones.
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#9The downside of this though is that unexpected positive results sometimes get buried. For instance, clinical trials have to report all adverse effects but not positive effects. We've had drugs tested for symptom X fail to have any effect, while causing bald people to have their hair grow back. Yet the company we were testing it for wasn't interested and those results never made it into the public domain.
i would assume that the positive results still get reported somewhere in the paper? so other people could still pick up on it in the future.
if they’re positive enough then the drug company will fund another study with that as the primary outcome. that seems prudent too: it should be the primary thing you’re examining so that you can design the study correctly, rather than a simple “oh and by the way” side note
Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See
#10I'm cynical and have no trouble believing that results were twisted immensely prior to the rule change in 2000. The incentives are huge: profit, prestige, or simply job security. I do wonder if this chart misrepresents something, though: there are studies that produce incidental--but genuinely valuable--discoveries. It's unclear to me if that accounts for the pre-2000 results or not. With the new rules, would there h…
You're right, the new rule doesn't filter out dishonest discoveries, only incidental ones (which may be dishonest). If you wanted to show benefit from an incidental discovery, you would have to run another study with that as its declared objective.
The author is kind of doing the same thing here that they accuse others of doing...