Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

motherjones.com

Post reply on HN
Print view View original

61–70 of 102 posts

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#61

Post by sudofail »

I have a general rule to ignore any statistical interpretation in articles like these. Data interpretation is hard. There are a lot of variables that need to be accounted for, and under the best circumstances of academic peer review, we still get it wrong.

This article also feels like it has an agenda. Maybe I'm not familiar with MotherJones, but the tone of the article strikes me as unprofessional. And the headline, that's obviously correct and means nothing. It's like saying, "Why You Shouldn't Trust Every Stranger You Meet".

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#62

Post by ineedasername »

qwerty456127 wrote: post #29

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…

It's not a counterproductive limitation. In fact it should lead to more research: In this type of clinical experiment, if your data happens to show a significant result that wasn't in the pre-planned outcomes, the proper process to follow is to formulate a second course of research designed around validating that new hypothesis.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#63

Post by xyzzyz »

qwerty456127 wrote: post #54
xyzzyz wrote: post #48

Earlier quoted context omitted.

"Significant" here doesn't mean what you think it means, but rather it has a technical meaning. Basically, it means that if the intervention has absolutely no effect, then there's less than 5% probability that measured "benefit" is just a random fluke -- thus, since you actually measured benefit, you are now left wondering whether the intervention actually has an effect, or you just happen to be in 5% of possible uni…

> since you actually measured benefit, you are now left wondering whether the intervention actually has an effect, or you just happen to be in 5% of possible universes where the randomness just happened to align this way. That still seems a good first step. It may also be curious to find out what were the things that have actually lead to the effect if not the one we were checking. E.g. it may happen that it the subs…

> It may also be curious to find out what were the things that have actually lead to the effect if not the one we were checking.

This sounds like a good idea in theory, but in practice, it is usually even harder to answer this question than the original question that yielded the data. This type of question requires extremely careful study designs, large sample sizes, accurate power estimates, etc.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#64

Post by BurningFrog »

ineedasername wrote: post #62
qwerty456127 wrote: post #29

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…

It's not a counterproductive limitation. In fact it should lead to more research: In this type of clinical experiment, if your data happens to show a significant result that wasn't in the pre-planned outcomes, the proper process to follow is to formulate a second course of research designed around validating that new hypothesis.

That's a nice theory. I worry that in reality a lot of those second research projects will never happen.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#65

Post by ineedasername »

superjan wrote: post #18

One could argue that this chart too is an example of post-hoc data mining. I still find the effect plausible, though.

Post-Hoc or not would depend on what hypothesis the researcher had before looking at the data. We don't really know, or at least I couldn't tell from the article.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#66

Post by profalseidol »

jmts wrote: post #47
geggam wrote: post #12

Earlier quoted context omitted.

Explain that to the layman in a way the entire Internet will understand. Until you can ... they are for all intents and purposes.

"Clinical trials are a type of scientific study. The studies shown in the article are of a particular type, and don't represent all studies, much like cows don't represent all farm animals. There may be similar cases across the remainder of science, but this case cannot be used to show that this is true, in the same way cows producing milk cannot be used to show that chickens produce milk. "A reasonable takeaway from…

Maybe we should have some good metric showing how small the sample size is compared to the whole. And maybe make it a rule to add 'Approximation' in the title if it uses that.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#67

Post by capnrefsmmat »

shoguning wrote: post #27
epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

Agreed, further: >every significant clinical study of drugs and dietary supplements for the treatment or prevention of cardiovascular disease between 1974 and 2012 There is room for a lot of bias when selecting which studies are 'significant' and on topic. Not to mention deciding which metric to report from the study.

Their procedure was carefully documented, including the inclusion criteria and how exactly they categorized the studies. They even had two teams independently search the literature to find studies meeting the criteria: https://journals.plos.org/plosone/article?id=10.1371/journal...

Literature reviews in meta-analyses are usually conducted like this, with specified lists of search keywords and flow charts of inclusion criteria.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#69

Post by brownbat »

In our world, it's considered malpractice if you have the option to include double blinding in your study design, but opt not to for convenience.

In some alternate universe, it is considered malpractice for those who design the study to be the same group that runs the study.

I don't think we can get there from here, but if we had a core track of theorists who designed studies, and a second equally prestigious track of practitioners, who independently tested and ran studies, experimental science would be much more rigorous.

Your prestige should be tied to your ability to identify novel experiments to try, or in rigorous testing procedures, never tied to your ability to shape data to make your claims appear grand.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#70

Post by L_Rahman »

function_seven wrote: post #7
smrtinsert wrote: post #5

Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.

Aren't they equivalent in their potential to allow post-hoc data mining? Whether it's a clinical trial, or a psychological study with n participants, or a look at how changes in x soil conditions affects the y tree. Anything where data samples are taken and statistically analyzed is prone to p-hacking and after-the-fact hypothesis changes. This example happens to be for a narrow set of studies: clinical trials for ca…

And in fact the replication crisis in psychology suggests that the practice goes beyond just clinical trials.
Post reply on HN