Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See • Hacker News Board

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#31

Post by jonathankoren » Thu, Nov 08, 2018, 10:16 PM UTC

cup wrote: post #6

The downside of this though is that unexpected positive results sometimes get buried. For instance, clinical trials have to report all adverse effects but not positive effects. We've had drugs tested for symptom X fail to have any effect, while causing bald people to have their hair grow back. Yet the company we were testing it for wasn't interested and those results never made it into the public domain.

That wouldn't happen. You would simply declare, that you want to check if the drug does cause hair growth, and then run a test explicitly looking for that.

Unless you're looking for the effect from the outset, you can't be sure that what you saw wasn't actually random. There's a term for what you're describing, it's p-hacking, and it's explicitly the very thing declaring what you're looking for before you run the test is designed to prevent.

Like many science related topics, there's a XKCD about p-hacking that describes a similar scenario to your example: https://xkcd.com/882/

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#32

Post by jacksnipe » Thu, Nov 08, 2018, 10:16 PM UTC

nemo44x wrote: post #14

Good science has its methods published, is reproducible and is peer reviewed. Another, perhaps better, headline would be: "Why You Shouldn't Trust Every 'Study' You See".

No True Scotsman. In fact, I’m pretty sure this is just another form of post-hoc reasoning.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#33

Post by amirmasoudabdol » Thu, Nov 08, 2018, 10:21 PM UTC

Although I agree with the argument but their chart and their data actually do not represent what they are claiming. I’ve checked their paper while back [1] and contacted them about it but haven’t heard anything back. Their argument is sound but only because!

[1]: https://amirmasoudabdol.name/likelihood-of-finding-null-effe...

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#34

Post by stilley2 » Thu, Nov 08, 2018, 10:21 PM UTC

epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

I would say that not all research needs to be held to the pre-registering standard of clinical trials. Some research is more exploratory, and should be interpreted that way.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#35

Post by fizzigy » Thu, Nov 08, 2018, 10:23 PM UTC

The article seems to suggest that scientific studies aren't reliable, but fails to point out the key takeaway that now (post registration requirement), studies are much more likely to be valid/useful than before.

So the headline really should read something like "Why you should trust scientific studies a lot more now than you did before"

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#36

Post by gmfawcett » Thu, Nov 08, 2018, 10:25 PM UTC

geggam wrote: post #12

smrtinsert wrote: post #5

Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.

Explain that to the layman in a way the entire Internet will understand. Until you can ... they are for all intents and purposes.

Hardly equivalent "for all intents and purposes." The purposes of the layman are nothing like those of the research community.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#37

Post by bgaragan » Thu, Nov 08, 2018, 10:28 PM UTC

qwerty456127 wrote: post #29

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…

I'm curious about this as well. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? It seems counterproductive indeed to waste time and money like that.

I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find.

Please correct me if I'm misguided on any of this.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#38

Post by bad_user » Thu, Nov 08, 2018, 10:30 PM UTC

epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

> With the advances in treatments for cardiovascular disease couldn't the problem just be harder?

I'm pretty sure we have no significant advances in the treatment of cardiovascular disease.

And given the alarmingly climbing rates of chronic diseases, e.g. cardiovascular disease, diabetes, obesity, cancer, I'm also pretty sure that the health care industry has done more harm than good when it comes to chronic diseases.

For everything else, for problems they can measure and treat with a pill, sure, I'm no anti-vaxxer, but they failed hard at chronic diseases, promoting cures and guidelines that did more harm than good.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#39

Post by abrichr » Thu, Nov 08, 2018, 10:31 PM UTC

qwerty456127 wrote: post #29

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…

The problem is that given enough data, you can always find patterns. Without a causal relationship, however, those patterns are likely spurious.

For a more in-depth analysis, see "The Deluge of Spurious Correlations in Big Data": https://www.di.ens.fr/users/longo/files/BigData-Calude-Longo...

Here's the abstract:

Very large databases are a major opportunity for science and data analytics is a remarkable new field of investigation in computer science. The effectiveness of these tools is used to support a ‘‘philosophy’’ against the scientific method as developed throughout history. According to this view, computer-discovered correlations should replace understanding and guide prediction and action. Consequently, there will be no need to give scientific meaning to phenomena, by proposing, say, causal relations, since regularities in very large databases are enough: ‘‘with enough data, the numbers speak for themselves’’. The ‘‘end of science’’ is proclaimed. Using classical results from ergodic theory, Ramsey theory and algorithmic information theory, we show that this ‘‘philosophy’’ is wrong. For example, we prove that very large databases have to contain arbitrary correlations. These correlations appear only due to the size, not the nature, of data. They can be found in ‘‘randomly’’ generated, large enough databases, which—as we will prove—implies that most correlations are spurious. Too much information tends to behave like very little information. The scientific method can be enriched by computer mining in immense databases, but not replaced by it.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#40

Post by ianbooker » Thu, Nov 08, 2018, 10:33 PM UTC

stilley2 wrote: post #34

epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

I would say that not all research needs to be held to the pre-registering standard of clinical trials. Some research is more exploratory, and should be interpreted that way.

That is right, And wrong.

There are explicit exploratory methods you can use to carry out research and that is very fine, but most research today is done using an inference model, where you formulate a hypothesis and use data to corroborate it.

That is one of the premises of big data that most people do not get: It enables you to do research explicitly exploratory again, while still being rigorous and somehow falsifiable..