Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To an alarming degree, science is not self-correcting

economist.com

Post reply on HN
Print view View original

121–130 of 145 posts

Re: To an alarming degree, science is not self-correcting

#121

Post by kybernetikos »

shanusmagnus wrote: post #38
tokenadult wrote: post #14

This is an excellent overview article on an important topic. It mentions many of the most influential authors on the topic of accuracy of scientific publications. Hacker News readers may enjoy "Warning Signs in Experimental Design and Interpretation"[1] by Peter Norvig, a LISP hacker who is now director of research at Google, on how to interpret scientific research. Norvig's essay is my all-time favorite link to shar…

This beautiful (parent) comment, in contrast to the main article, actually contains links and references to the items it refers to. Is there some compelling reason I don't understand why _so_ many magazines / websites, even high profile reputable ones, cannot manage to link or cite?

I find this particularly galling since many of these publications have fact checkers who have gone to the effort of identifying claims in the original article, tracking down the evidence to prove/disprove it and checking that the article doesn't materially misrepresent the evidence.

This is the internet, it shouldn't cost any more to let me see the fact checkers report, and it will save me a massive amount of time reproducing their work. Expandable margin notes on a per paragraph basis would be perfect.

Re: To an alarming degree, science is not self-correcting

#122

Post by sungx105 »

thisisnotatest wrote: post #120

How about a karma/bittorrent system for researchers to encourage replication? In such a system, the "application fee" for submitting your experiment to a journal would be that you must also replicate someone else's experiment that was also submitted to the journal. This system could scale arbitrarily.

This would not be a realistic solution, because the time it can take to do experiments might be an extremely long time. This would make it so that everyone would be (at best) half as productive as before, since they would have to spend the time necessary to carry out the replication. This would only be worsened if what you were required to replicate was not in a similar area to the researcher's area of expertise.

Re: To an alarming degree, science is not self-correcting

#123

Post by olalonde »

guelo wrote: post #62
capnrefsmmat wrote: post #45

I am sometimes amused to read failed replication studies which do not calculate their own statistical power -- so they do not know whether, assuming the original study was correct, they will have the sample size to reliably detect an effect. A failed replication then says nothing about the original conclusions. Statistical error is a hobby of mine. The Economist mentions a few pervasive errors, but there are many mor…

Nice article, but copying an xkcd image without copying the image's tooltip text is criminal.

Which is: "'So, uh, we did the green study again and got no link. It was probably a--' 'RESEARCH CONFLICTED ON GREEN JELLY BEAN/ACNE LINK; MORE STUDY RECOMMENDED!'"

Re: To an alarming degree, science is not self-correcting

#124

Post by XorNot »

vortmeester wrote: post #32
pnathan wrote: post #13

In my experience at the university performing graduate research - * Newness is prized * Replicating old science is not prized * Funding and papers happens for breakthroughs In software engineering - * many bugs in code creep through exacting peer reviews. Recapping the short list: new things make you money, reviewing things is error prone, replicating old things doesn't make you money until someone wants to rely on t…

As an experimental physicist, I want to point out that negative results do get published -- for example, much of the experimental work on gravity amounts to the showing of no measurable difference from theoretical expectation, to a high degree of accuracy [1] -- but writing the papers for them is much harder than writing up positive results. The reason is simple: you have to convincingly show that your failure to sho…

Exactly this. Tons of people would love to publish negative results. But when you're in the situation with something not working, the actual mechanics of proving it doesn't work is a much harder problem.

Re: To an alarming degree, science is not self-correcting

#125

Post by XorNot »

pdonis wrote: post #101
lamontcg wrote: post #98

Earlier quoted context omitted.

They looked at siting effects as well: http://www.scitechnol.com/2327-4581/2327-4581-1-107.php They used Watts own siting data and showed that both poor and good sites showed the same temperature trend over time. The data back to 1860 are also sufficient to establish the current warming trend and through other analysis (e.g. Meehl, et al's work) to establish the climate sensitivity. Climate science as a discipline do…

Thanks for the link, I'll take a look at it. The data back to 1860 are also sufficient to establish the current warming trend Only if you assume that there are no significant cyclic effects on longer time scales. Climate science as a discipline does not hinge on knowing exactly how warm it was 250 years ago within +/- 0.5C. It does if you're claiming, as the IPCC does based on the input of climate scientists, that te…

This is not as relevant as you're implying - if there are cyclic effects on long time scales, you're implying they're highly exponential - long periods of no warming, and then sudden periods of rapid warming (oddly correlated with the rise in human CO2 emissions at that).

There's no postulated mechanism for how this could be the case - where is the extra energy or heat retention coming from and managing to come into play so suddenly since the 1970s?

Whereas, if one accounts for anthropogenic CO2, it explains it superbly.

Re: To an alarming degree, science is not self-correcting

#126

Post by einhverfr »

Alex3917 wrote: post #88
tokenadult wrote: post #14

This is an excellent overview article on an important topic. It mentions many of the most influential authors on the topic of accuracy of scientific publications. Hacker News readers may enjoy "Warning Signs in Experimental Design and Interpretation"[1] by Peter Norvig, a LISP hacker who is now director of research at Google, on how to interpret scientific research. Norvig's essay is my all-time favorite link to shar…

> Warning Sign I5: Taking p too Seriously What does it say about X when you have 10 studies saying X is true with p Also, while I do like norvig's essay, all the errors he goes through are fairly pedestrian. If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. (Assuming researchers researching researchers are more trustworthy than researchers.)

> If you read the modern scientific literature on research, the rabbit hole goes vastly deeper.

On indeed. Heisenberg used to say, "data does not imply theory."[1] Once you deeply internalize that, you realize you can't really be that, ahem, certain about anything in scientific thought outside of the limited domain of how scientific truth is defined (namely that the model has predictive value, which is entirely different from whether it has ontological value).

[1] http://archive.org/details/PhysicsPhilosophy

Re: To an alarming degree, science is not self-correcting

#127

Post by SeanDav »

I suppose it is naive to believe that scientific papers should not be published until they are peer reviewed and more importantly, their results duplicated.

If this is impractical there should be separate sections in journals for non-replicated and replicated results or perhaps there needs to be separate tiers, with something like the "Gold Tier" only containing papers that have had their results replicated multiple times in multiple countries.

Probably wishful thinking, but it seems clear, something needs to change.

Re: To an alarming degree, science is not self-correcting

#128

Post by skylan_q »

einhverfr wrote: post #126
Alex3917 wrote: post #88

Earlier quoted context omitted.

> Warning Sign I5: Taking p too Seriously What does it say about X when you have 10 studies saying X is true with p Also, while I do like norvig's essay, all the errors he goes through are fairly pedestrian. If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. (Assuming researchers researching researchers are more trustworthy than researchers.)

> If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. On indeed. Heisenberg used to say, "data does not imply theory."[1] Once you deeply internalize that, you realize you can't really be that, ahem, certain about anything in scientific thought outside of the limited domain of how scientific truth is defined (namely that the model has predictive value, which is entirely diffe…

Try telling that to economists who claim to discover macroeconomic theories through econometric models and call it "empirical"

Re: To an alarming degree, science is not self-correcting

#129

Post by [deleted] »

tokenadult wrote: post #14

This is an excellent overview article on an important topic. It mentions many of the most influential authors on the topic of accuracy of scientific publications. Hacker News readers may enjoy "Warning Signs in Experimental Design and Interpretation"[1] by Peter Norvig, a LISP hacker who is now director of research at Google, on how to interpret scientific research. Norvig's essay is my all-time favorite link to shar…

[deleted]

Re: To an alarming degree, science is not self-correcting

#130

Post by unhammer »

shanusmagnus wrote: post #38
tokenadult wrote: post #14

This is an excellent overview article on an important topic. It mentions many of the most influential authors on the topic of accuracy of scientific publications. Hacker News readers may enjoy "Warning Signs in Experimental Design and Interpretation"[1] by Peter Norvig, a LISP hacker who is now director of research at Google, on how to interpret scientific research. Norvig's essay is my all-time favorite link to shar…

This beautiful (parent) comment, in contrast to the main article, actually contains links and references to the items it refers to. Is there some compelling reason I don't understand why _so_ many magazines / websites, even high profile reputable ones, cannot manage to link or cite?

Science Exchange: https://www.scienceexchange.com/reproducibility

Ioannidis' article on false pos/neg and stat.power: http://www.plosmedicine.org/article/info:doi/10.1371/journal...

Bohannon's "Who's afraid of a little peer review": http://www.sciencemag.org/content/342/6154/60.full (fake submissions)

Also related I think is this: http://neurotheory.columbia.edu/~ken/cargo_cult.html ;-)

Post reply on HN