Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

Post reply on HN
Print view View original

21–30 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#21

Post by cs702 »

My first reaction to this headline was "duh." Of course we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims!

My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the prestigious institution in which he works, he is coming across people who treat science like religion.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#22

Post by timwaagh »

studentrob wrote: post #19
gnuvince wrote: post #6

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

most cs papers i read (actually all) do -not- include code (or input data). which makes this an issue even in the case of cs.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#23

Post by wccrawford »

ehnto wrote: post #17

We learned the importance of this in high school science and it baffles me that it's not already the case.

We have some kind of weird hero-worship of scientists where the general public just believes what they say, even if they never even attempt to replicate their results. They do an experiment (which may or may not be scientifically sound to start with) and then publish results, and the public eats it up.

And then people have the nerve to say, "Last week chocolate was bad for me, now it's good? Make up you mind!" No, stop listening to un-replicated studies! Jeez.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#24

Post by mannykannot »

studentrob wrote: post #19
gnuvince wrote: post #6

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

Rerunning code does not count as reproduction as it is not an independent test of what is being claimed. You do have the option of verifying the code against the goals of the study, though that is a review rather than a reproduction (which is still valuable.)

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#25

Post by mablap »

appleflaxen wrote: post #15
undergroundOps wrote: post #7

I'm a physician and I've been suggesting this to my colleagues for a few years, only to be met with alienated stares and labeled cynical. The doctors and doctors-in-training I work with have altruistic motives, but place too much stock in major medical studies. They also frequently apply single-study findings to patient care, even to patients that would've been excluded from that study (saw this a lot with the recent…

but in medicine, what is the practical alternative? how do you incorporate these findings? ignore them? if so, it's probably bad for your patients. the only thing worse than a single-study finding is a zero-study finding.

No. Step 1: Do no harm. In practice, this means staying up to date on the latest research (by law), but follow the guidelines. These are updated regularly, just not after each and every study that comes out. And it's a good thing a lot of doctors exercise caution.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#26

Post by maze-le »

studentrob wrote: post #19
gnuvince wrote: post #6

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

Not necessarily. There are fields like complexity and computability wich are pureley mathematical in nature and have little to do with actual code, but much with the abstract theoretical fundamentals of code in general. And on the other hand you have fields like machine learning, where you can have an algorithm, wich you can implement (or have as code), but you don't nessessarily know the values of certain parameters, specific for your problem space.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#27

Post by mathgeek »

varjag wrote: post #16
nxzero wrote: post #9

Earlier quoted context omitted.

Real issue is that the way science is funded is broken.

Broken assumes it was working well at some point. Nothing new here.

It worked pretty well back when scientific inquiry was funded by selling inventions or other work that the scientist did "as a day job." That was a long time ago, though.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#28

Post by mablap »

undergroundOps wrote: post #7

I'm a physician and I've been suggesting this to my colleagues for a few years, only to be met with alienated stares and labeled cynical. The doctors and doctors-in-training I work with have altruistic motives, but place too much stock in major medical studies. They also frequently apply single-study findings to patient care, even to patients that would've been excluded from that study (saw this a lot with the recent…

Personal experience (I've met many pre- and current doctors during my studies) suggests that few doctors are scientists. They know a lot of things, but they're not what I'd call scientists in the same way a physicist or biologist would be (and I've often wondered why they'd get a "Doctorate" at all). So this might explain the abovementioned behaviour...

Yet I'm surprised by your colleagues behaviour nonetheless. I would have thought they'd have more retenue.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#30

Post by nxzero »

cs702 wrote: post #21

My first reaction to this headline was "duh." Of course we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims! My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the…

Of course you as a reader of said claims confirm at the very least that they've been independently reproduced, right?

(If so, this shouldn't be news.)

Post reply on HN