Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

motherjones.com

Post reply on HN
Print view View original

81–90 of 102 posts

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#81

Post by dbt00 »

epistasis wrote: post #78
prakashk wrote: post #16

Earlier quoted context omitted.

> Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? > However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data…

I don't see that chart supporting a sudden change. The time point could be moved quite a bit and tell the same story. Also I had not heard of Kevin Drum before this, and had a positive view of Mother Jones. I'm left with a poor impression of Kevin Drum and a hit to Mother Jones' reputation after reading this.

You could go read the original paper and examine the actual data and scientific conclusions drawn from the underlying data as represented on that chart.

You could look for confirmation or disconfirmation of the hypothesis that preregistration leads to an increase in null results in other data sets.

Both of those seem like more useful ways to dispute this finding than squinting at a graph and finding fault with a blog post headline.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#82

Post by Natsu »

cinquemb wrote: post #79
Natsu wrote: post #57

Earlier quoted context omitted.

> The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.

>Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc. In my experience working with/contracting for neuro labs, a lot of researchers don't really know how to fully leverage the technology that's available, and often rely upon proprietary tools they have limited knowledg…

I think it's one of those things where a few point shift in the average of the distribution really changes some numbers near the edges of it, though.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#83

Post by epistasis »

dbt00 wrote: post #81
epistasis wrote: post #78

Earlier quoted context omitted.

I don't see that chart supporting a sudden change. The time point could be moved quite a bit and tell the same story. Also I had not heard of Kevin Drum before this, and had a positive view of Mother Jones. I'm left with a poor impression of Kevin Drum and a hit to Mother Jones' reputation after reading this.

You could go read the original paper and examine the actual data and scientific conclusions drawn from the underlying data as represented on that chart. You could look for confirmation or disconfirmation of the hypothesis that preregistration leads to an increase in null results in other data sets. Both of those seem like more useful ways to dispute this finding than squinting at a graph and finding fault with a blog…

I'm not sure if you meant to reply to me, because every aspect of your post is wrong.

The "squinting at the graph" was assuming that there's a sudden change. I read the paper, and looked up several studies and came to the conclusion that the paper was being misrepresented by the blog text. And I agreed with the headline, but not with the blog text.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#84

Post by KineticLensman »

saalweachter wrote: post #22
epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

I'll go one stronger and say "You Shouldn't Trust Any One Scientific Study You See". Individual studies can be really interesting. They're important for researchers to know about to inform their future work. But any one study - even ones that are done honestly, with good methodology and sound foundations - can be just totally wrong. There could be confounding factors you couldn't have known about that completely inva…

Yes. Science can be an adversarial process as people converge on the truth. Like a legal trial, you shouldn't take out of context any single statement from a defence or proceution lawyer as representative of the whole truth

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#85

Post by icelancer »

"Before 2000, researchers cheated outrageously. They tortured their data relentlessly until they found something—anything—that could be spun as a positive result, even if it had nothing to do with what they were looking for in the first place. After that behavior was banned, they stopped finding positive results. Once they had to explain beforehand what primary outcome they were looking for, practically every study came up null. The drugs turned out to be useless."

This is a ridiculous "plain English" description of what is happening here, and I say that as someone who is regularly very critical of academia, drug trials, and research science (I've lived it; check my bio).

Clinical trials and mandatory registration are great things. It does not mean that researchers were massively cheating in the past, however - it means that they were finding secondary and tertiary findings and reporting them instead of the main investigative thrust of the research. Yes, some blatant cheating happened, as did p-hacking (though this problem still exists), but to act like the clinical registration database completely stopped a massive ring of fraud is ridiculous and the data does not support that, merely a conspiratorial narrative around it.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#86

Post by icelancer »

epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

>> In any case, don't trust every popular news article you read about science, particularly if it's written by Kevin Drum and posted on Mother Jones.

Very good advice. Over 95% of the crap posted on Mother Jones should be ignored.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#87

Post by pimmen »

Fomite wrote: post #74
pimmen wrote: post #73

Earlier quoted context omitted.

Possible sources of skew is more interesting. Larger, skewed samples can be far worse than smaller, randomized ones.

This is the source of a classic exam question given by Ken Rothman, an epidemiologist: As sample size goes up, the probability that an estimate's confidence interval contains the true value of an effect goes.. A) Up B) Down

Thanks for the reference! Very interesting Twitter discussion that sparked. :)

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#88

Post by gaius »

L_Rahman wrote: post #70
function_seven wrote: post #7

Earlier quoted context omitted.

Aren't they equivalent in their potential to allow post-hoc data mining? Whether it's a clinical trial, or a psychological study with n participants, or a look at how changes in x soil conditions affects the y tree. Anything where data samples are taken and statistically analyzed is prone to p-hacking and after-the-fact hypothesis changes. This example happens to be for a narrow set of studies: clinical trials for ca…

And in fact the replication crisis in psychology suggests that the practice goes beyond just clinical trials.

The replication crisis in every field of science. Two-thirds of reported results cannot be reproduced.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#89

Post by gaius »

gmfawcett wrote: post #36
geggam wrote: post #12

Earlier quoted context omitted.

Explain that to the layman in a way the entire Internet will understand. Until you can ... they are for all intents and purposes.

Hardly equivalent "for all intents and purposes." The purposes of the layman are nothing like those of the research community.

The purposes of the layman are nothing like those of the research community.

The "laymen" are the ones funding all of this through their taxes. I'm sure scientists would love it if the plebs just shut up and kept on giving them money, but it doesn't work like that. In fact that is why it is becoming harder and harder for scientists to communicate with the public to affect policy.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#90

Post by teddyh »

nostrademons wrote: post #26
clircle wrote: post #4

Earlier quoted context omitted.

I don't follow this comment. What would you plot on the vertical axis?

Clickthroughs, presumably. I assume that taruz is saying that journalists should be required to say what they're investigating before publishing an investigative report. Right now, they start investigating, and if there's something outrageous (even if it wasn't what they were initially looking for), they publish. Sometimes they even skip the "start investigating" part and just put up a tip-line for anyone who has a b…

There’s also this to consider:

“If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him.”

– Commonly attributed to Cardinal Richelieu

I.e. any fact or action can be willfully misinterpreted to fit almost any narrative.

Post reply on HN