Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

motherjones.com

Post reply on HN
Print view View original

71–80 of 102 posts

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#71

Post by nkurz »

bgaragan wrote: post #37
qwerty456127 wrote: post #29

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…

I'm curious about this as well. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? It seems counterproductive indeed to waste time and money like that. I'm having…

I presume you are being genuine, but you should know that your comment is almost indistinguishable from satire. More specifically (and hopefully more helpfully) it sounds like the comments made by Brian Wansink before his recent fall. Google for his story if you are unfamiliar.

Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing?

Assuming that the effects are legitimate is exactly the problem. You are right, if we somehow know that the effects found are real and reproducible, then all is good. The problem is that we almost never know this. Presumably what you mean is "If the results are reproducible, what's the harm?". I'd agree with this, but the problem is how to know ahead of time that the results are going to be reproducible.

Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result?

If they did the statistics correctly (accounting for the multiple inferences, all assumptions about iid data met, no biased dropouts, everything else aboveboard) and got a really solid result, then yes, it's theoretically likely that the results would be reproducible. The problem is that they almost certainly didn't do the statistics correctly, and intentionally or not they probably violated a lot assumptions. In too many cases, even the main line conclusions can't be replicated. It's rarely "safe" to assume that an effect is real until it's actually been replicated, and almost never safe to make this assumption for result obtained by sifting the data after the fact looking for correlations.

I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find.

There are ways to do it this way, but it usually takes larger sample sizes than are available. It also requires "bespoke" statistics that are easy to get wrong. In practice, it's usually better to use the incidental "results" as idea generators for future experiments, rather than assuming that the findings are real and don't require further testing.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#72

Post by lolc »

qwerty456127 wrote: post #50
monktastic1 wrote: post #45

Earlier quoted context omitted.

Are you familiar with the multiple comparisons problem[1]? The problem is that in any sufficiently rich dataset, you can find something unusual-looking, and if you're not straightforward about how much digging you had to do to find it, it will look much more special than it actually is. So if a benefit is found by chance, that's fine, but it should only motivate further research which specifically look for that effec…

> but it should only motivate further research which specifically look for that effect Indeed. But this doesn't disqualify looking for "something" as a valid and useful method of research to be a stage of the whole research chain. That ought to be allowed although research papers produced this way should make this clear.

It's called exploratory research and everybody is fine with it when it's labeled as such.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#73

Post by pimmen »

profalseidol wrote: post #66
jmts wrote: post #47

Earlier quoted context omitted.

"Clinical trials are a type of scientific study. The studies shown in the article are of a particular type, and don't represent all studies, much like cows don't represent all farm animals. There may be similar cases across the remainder of science, but this case cannot be used to show that this is true, in the same way cows producing milk cannot be used to show that chickens produce milk. "A reasonable takeaway from…

Maybe we should have some good metric showing how small the sample size is compared to the whole. And maybe make it a rule to add 'Approximation' in the title if it uses that.

Possible sources of skew is more interesting. Larger, skewed samples can be far worse than smaller, randomized ones.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#74

Post by Fomite »

pimmen wrote: post #73
profalseidol wrote: post #66

Earlier quoted context omitted.

Maybe we should have some good metric showing how small the sample size is compared to the whole. And maybe make it a rule to add 'Approximation' in the title if it uses that.

Possible sources of skew is more interesting. Larger, skewed samples can be far worse than smaller, randomized ones.

This is the source of a classic exam question given by Ken Rothman, an epidemiologist:

As sample size goes up, the probability that an estimate's confidence interval contains the true value of an effect goes..

A) Up B) Down

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#75

Post by Fomite »

whack wrote: post #49

Is it possible that prior to 2000, the studies that found a null result, were simply not published? Hence why they are underrepresented in that chart? My understanding is that it's very hard to get null results published in scientific journals, though I'm not sure if that applies to clinical studies as well.

One interesting effect that pre-registration has is that there are now several journals that allow you to publish your protocol, and agree to publish the followup paper about the results regardless of the outcome.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#76

Post by qyz721 »

I often wonder about the kind of research done in computer science--how much is it influenced by the kinds of things that easily get you a Ph.D., versus the kinds of things that are useful but less apparently flashy.

A lot of the PL students at my school are extremely wary of doing any follow-up work on ideas that have been published before, even if the implementations of those things are obviously shoddy and don't really demonstrate that the idea works. There's a lot of novelty chasing, which is part of what's pushing people to include deep learning in their work, since it often allows them to claim novelty, even if their results aren't very good.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#77

Post by metaphor »

geggam wrote: post #58
radus wrote: post #52

Earlier quoted context omitted.

Do you apply this reasoning to articles or comments discussing kernel development? Some topics have higher irreducible complexity than others. More importantly, I can make whatever I want public, and I can do so with whatever audience I have in mind.

Interesting. So inciting panic online is fine but not in a theater ?

Your remark finds difficulty in satisfying clear and present danger.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#78

Post by epistasis »

prakashk wrote: post #16
epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

> Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? > However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data…

I don't see that chart supporting a sudden change. The time point could be moved quite a bit and tell the same story.

Also I had not heard of Kevin Drum before this, and had a positive view of Mother Jones. I'm left with a poor impression of Kevin Drum and a hit to Mother Jones' reputation after reading this.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#79

Post by cinquemb »

Natsu wrote: post #57
prakashk wrote: post #16

Earlier quoted context omitted.

> Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? > However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data…

> The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.

>Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.

In my experience working with/contracting for neuro labs, a lot of researchers don't really know how to fully leverage the technology that's available, and often rely upon proprietary tools they have limited knowledge of, which doesn't bode well for being able explore for themselves.

The few that I have met that can push the limits of current technology are working in labs ran by the above…

I'm not sure how it is in other fields, but in convos from some other commentators on HN over the past years, makes me think this is not just in neuroscience.

Maybe the problem is that the skills needed to explore the solution space and communicate it effectively have gone up because the complexity it has added to the process without research labs/academia addressing the gap sufficiently? I don't think this is a problem with just labs or academia though, not many people in general have the skills to be able to leverage technology to it's fullest for even the most banal tasks.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#80

Post by TangoTrotFox »

epistasis wrote: post #8

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

Problems getting harder is not really a valid explanation for the current state of science, or any particular field. At any given glimpse of society, their current state of the art is the best they've managed with all they know. Further progress is always harder, even when we now think of it as something that should have been easy. For instance the basic concept that things happen for a reason is something everybody understands now, even only a completely intuitive level - yet it was one of the breakthroughs of the Greeks. And Newtonian mechanics is also something people tend to understand quite intuitively, yet following the Greeks it would take the better part of two thousand years for people to consider that e.g. objects in motion stay in motion unless acted on by something else. Artistotlean mechanics held that continued motion required continued force. Artistotle seemingly did not consider friction, air resistance, and so on that cause objects to slow. Or just consider the tens to hundreds of thousands of years to develop writing, or to enter into the iron age, and so on.

There's some argument to be said that in times past there were fewer "scientists" working on these topics, explaining the delays. But I don't think that's fair. The reason I put "scientist" in quotes is because many of these things have minimal prerequisites and the average home would have all that's needed to experimentally test and discover these concepts. And their immense value, far beyond the academic, meant their discovery likely would have been able to rapidly spread regardless of the origin - so that, at least to some degree, precludes authority as a necessity.

Many things that now constantly elude our ability to understand, perhaps dark matter is a great example, will likely one day be child's play.

Post reply on HN