Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

11–20 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#11
The big issue right now is funding of replicated research, who wants to fund a research to prove someone else was right? Most of these funds are granted based on potential outcome of the new discovery like: potential business, patents, licenses, etc... not being the first one would probably wipe most of these benefits, cutting down to a small probably getting funded...

Now, straight to the point, who's going to pay for the repeated research to prove the first one?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#12
post #2

Totally agree. I'd go even further and make free licenses on scientific source and datasets mandatory. Research that is funded by public money should lead to public code and data.

What about defense research?

Well, there's the GPL-styled approach: anyone with access to the results must also have access to the associated data. This doesn't mean it is mandatory to make it public, though you'd have to restrict the redistribution freedom.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#14
Sometimes in cs if your research is embedded in a huge ecosystem, it can become quite expensive to reproduce results. I mean proper reproduction, not just rerunning the Benchmarks. If you are dealing with complicated stuff, the reproducer might also just not be able to do the same thing technically.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#15

I'm a physician and I've been suggesting this to my colleagues for a few years, only to be met with alienated stares and labeled cynical. The doctors and doctors-in-training I work with have altruistic motives, but place too much stock in major medical studies. They also frequently apply single-study findings to patient care, even to patients that would've been excluded from that study (saw this a lot with the recent…

but in medicine, what is the practical alternative?

how do you incorporate these findings? ignore them?

if so, it's probably bad for your patients. the only thing worse than a single-study finding is a zero-study finding.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#16
post #9
post #6

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

Real issue is that the way science is funded is broken.

Broken assumes it was working well at some point. Nothing new here.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#18
post #13

I thought this was in the definition of "scientific"

Peer review is how most science is defined as science and peer review does not require reproduction of the work.

Much of what we peer review is not real science, at least in its definition of applying the scientific method.

For example, much of computer "science" is not. Math maybe, engineering probably, design sometimes, but "science" is rarely done. BUT the science envy is there, especially post 1990s, and it is as confusing as heck when multiple definitions of "science" collide in a conference culture.

Yes I'm a researcher, no I'm not a scientist.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#19
post #6

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#20
post #11

The big issue right now is funding of replicated research, who wants to fund a research to prove someone else was right? Most of these funds are granted based on potential outcome of the new discovery like: potential business, patents, licenses, etc... not being the first one would probably wipe most of these benefits, cutting down to a small probably getting funded... Now, straight to the point, who's going to pay f…

Perhaps those who are skeptical of the first research (and are losing money from it) should fund the replication research.

Incentive for corrupting the data seems high, however.

Post reply on HN