Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

I'll go one stronger and say "You Shouldn't Trust Any One Scientific Study You See".

Individual studies can be really interesting. They're important for researchers to know about to inform their future work. But any one study - even ones that are done honestly, with good methodology and sound foundations - can be just totally wrong. There could be confounding factors you couldn't have known about that completely invalidate the result. Your test subjects could be unusual in some way, your animal models could be a poor analogue for humans in this particular case, you could have just had really aberrant statistical flukes in your statistic sampling.

It's the body of scientific research, the dozens, hundreds, thousands of studies stacked on top of each other that bring certainty.

This would seem to indicate that the results of drug research are unpredictable. Therefore, if you are required to predict your results, you are set up to fail.

My question is: Was the free-range research actually effective? Or, was it "technically-correct effective just so you can't call me out on a failure, moving on..."

Clickthroughs, presumably.

I assume that taruz is saying that journalists should be required to say what they're investigating before publishing an investigative report. Right now, they start investigating, and if there's something outrageous (even if it wasn't what they were initially looking for), they publish. Sometimes they even skip the "start investigating" part and just put up a tip-line for anyone who has a beef to get a story out there.

This is great for manufacturing outrage and hence clicks, but it gives the public a hugely skewed perspective on how the world is. Imagine that 0.01% (1 in 10,000) of all peoples' actions are outrageous and will piss off a large portion of planet. Most people, by those numbers, would say that the majority of folks are decent, law-abiding citizens. Now imagine that a news outlet is allowed to freely go over someone's life, and they end up evaluating 1000 actions. There's a 10% chance that they'll find something outrageous. Now imagine that 10 such reporters do this to 10 people, and if any one of them finds something publishable (= outrageous), they go to press. Suddenly there's a 64% chance that one of them will find something, and you've likely got your news cycle for the day.

With the millions of people looking for something bad that a tip-line can generate, outrage is virtually assured. And that's where journalism is today. The world isn't actually a worse place than it was in 1980; in fact, by most metrics it's significantly better. But we've increased the amount of unpleasantness that people can be exposed to by 3-4 orders of magnitude and then implemented a selection filter that ensures that only the worst stories go viral. Of course we get only bad news; that's all that's economically viable, and we have such a large sample size that we can surely find it.

I'd have to agree with this; every news outlet tries to treat a lack of proper controls as a breaking story / massive conspiracy theory to get a buzz going around the topic. I think it's important to inform people but unfortunately, most people would rather buy into the idea that there was malicious intent rather than also evaluating the possibility that, like most evolving business areas, it takes time to build in the proper procedures to attain measurably objective results. This happens in every industry and isn't unique to science, however, in this case, the physicians recommending these drugs to their patients should have been able to read between the lines or at the very least have a managed treatment plan to evaluate the efficacy on those they're treating.

I think you meant “shouldn’t”.