Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See • Hacker News Board

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#11

Post by scotty79 » Thu, Nov 08, 2018, 9:35 PM UTC

caymanjim wrote: post #3

I'm cynical and have no trouble believing that results were twisted immensely prior to the rule change in 2000. The incentives are huge: profit, prestige, or simply job security. I do wonder if this chart misrepresents something, though: there are studies that produce incidental--but genuinely valuable--discoveries. It's unclear to me if that accounts for the pre-2000 results or not. With the new rules, would there h…

> With the new rules, would there have to be another study stating the new objective?

I think it's only fair to force you to replicate at least once the positive result you think you see in the data you collected for another purpose before you can claim you got something.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#12

Post by geggam » Thu, Nov 08, 2018, 9:38 PM UTC

smrtinsert wrote: post #5

Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.

Explain that to the layman in a way the entire Internet will understand.

Until you can ... they are for all intents and purposes.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#13

Post by certmd » Thu, Nov 08, 2018, 9:43 PM UTC

While many on this forum may understand why what this analysis shows is not good science, the article does a poor job of explaining why finding positive results one wasn't originally looking for may not be reliable.

I can imagine a non-statistically minded person thinking "So what it's not what they were looking for originally? We're missing positive findings now. This is a terrible regulation." When in reality these "positive" findings were p-hacked to meet minimum criteria for statistical significance and likely arose by chance since luck would have it that in any study there will be some set of data that by chance is a statistical aberration.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#14

Post by nemo44x » Thu, Nov 08, 2018, 9:44 PM UTC

Good science has its methods published, is reproducible and is peer reviewed.

Another, perhaps better, headline would be: "Why You Shouldn't Trust Every 'Study' You See".

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#15

Post by Udik » Thu, Nov 08, 2018, 9:52 PM UTC

smrtinsert wrote: post #5

Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.

If anything, clinical trials should be more reliable than the average scientific study: after all they are experiments, with well defined numbers (sample size, effects measurement, controlled conditions). Compare with all non-experimental science- including for example most environmental and climate science, where, if experiments are made at all, the results are wildly extrapolated and generalized.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#16

Post by prakashk » Thu, Nov 08, 2018, 9:52 PM UTC

epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

> Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder?

The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before?

> However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at.

Is it possible that your own preconceived notions about the author and the publication may have caused you to judge this way?

In any case, don't trust every comment posted on Hacker News (including my own) :)

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#17

Post by DrNuke » Thu, Nov 08, 2018, 9:53 PM UTC

From a science perspective, positive results are overrated; the most important aspect is to investigate a “good” question. Researchers understand “good” has not the same meaning everywhere or to every stakeholder. More at large, blurred journalism is attacking science to undermine the scientific method, aka meaningful questions + proper scrutiny + full transparence. This is a sign of the times, though.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#18

Post by superjan » Thu, Nov 08, 2018, 9:54 PM UTC

One could argue that this chart too is an example of post-hoc data mining. I still find the effect plausible, though.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#19

Post by rossdavidh » Thu, Nov 08, 2018, 10:00 PM UTC

taruz wrote: post #2

I want a chart like that but for journalism.

Good idea, but you'd have to have an objective way of deciding how accurate the article was.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#20

Post by rossdavidh » Thu, Nov 08, 2018, 10:03 PM UTC

While some have pointed out that this very article could be considered post-hoc data mining, I think it is not. This exact kind of affect was what was intended by the requirement to pre-register. Looking afterwards to see if the intended affect (change in how many studies report benefits) appears to have happened in reality, makes perfect sense. They didn't institute the requirement at random, and then later consider that maybe it could have an impact of reducing (perhaps spurious) findings of significant benefit; that was more or less the only reason for doing so.