Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

Post reply on HN
Print view View original

71–80 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#71

Post by BaronSamedi »

cs702 wrote: post #21

My first reaction to this headline was "duh." Of course we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims! My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the…

My reaction was the same. I would argue that the requirement is even stronger: the hypothesis must be verified by different scientists over time. One of the key roles of the scientific method is error correction for bugs in our mental machinery (i.e., cognitive biases, of which there are many ).

I look at science as an essentially evolutionary process: a single study has a very low probability of being accurate, if the results survive additional testing over time, the probability of accuracy increases.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#72

Post by dalke »

eru wrote: post #61
dalke wrote: post #54

Earlier quoted context omitted.

So, $1 of public funding triggers public code release? Or is there a threshold? No one has been able to tell me why the need for reproducibility requires software freedom. Consider the program 'nauty'. It is available in source code form for anyone to review, but it cannot be used for military purposes. That's not free, certainly. But isn't that enough to call it good science? Similarly, consider the clause "only for…

> So, $1 of public funding triggers public code release? Or is there a threshold? Yes, 1$, no threshold necessary. For projects where the cost of publication of data is not worth the dollar, people will just not accept the dollar. (That's where a 'natural threshold' comes in.)

So, if I have a summer intern who is paid by the government, and helps on my otherwise non-government-funded project, then is that enough to trigger the requirement?

Or, if I use a government facility, like a supercomputing center, then that also trigger the release requirement? Even if no money changes hands? What about a government network?

It sounds very tricky. If I get a government grant to work on a project, and that grant buys equipment X, which I also use for another project, then must both projects be subject to this form of release?

Even if the second project is 10 years later?

If in working on project X, but in the process of working on X I find some new knowledge Y, and publish it, even though that has nothing to do with the grant from the original X, does that count? (Think AT&T's observation of the cosmic microwave background, when their goal was to reduce noise in terrestrial microwave communications.)

It seems like a very complicated scheme.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#73

Post by hdra »

thefastlane wrote: post #55
mydpy wrote: post #53

I agree in principle. There are a few concerns: 1. How should we receive costly research that took special equipment and lots of time to develop and cultivate? I.e., CERN? 2. A lot of research is published, ignored, and then rediscovered. In this case, we may want to accept the research until it cannot be repeated (i.e., in another journal publication). 3. Reviewers of academic publications probably are not qualified…

"Isn't the academic system at its core kinda... broken?" can your elaborate on what you mean?

I gather its things like misaligned incentives. Like funding sources dictating the "desirable" result, reluctance to publish negative results, pursue of (vanity) metrics that leads to quota system by universities, etc.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#74

Post by crispyambulance »

gnuvince wrote: post #6

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

I don't think reproducibility should necessarily mean literally repeating the same study over again.

Whether or not a paper is fully correct is less important that the further work it stimulates or informs-- either via more papers (impact factor) or via APPLICATIONS of the research.

In the example you're giving, to get accepted, the author doesn't need to do much more than extend the research in some small way, compare it to other research, or explore some aspect more fully. If someone is going to take the time to repeat something, they might as well go a little further.

If someone takes the time to apply research results they're going to, in a way, test the validity of the results. Maybe the author's experiences in the pharma world is very different, but I doubt that.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#75

Post by adrianN »

studentrob wrote: post #19
gnuvince wrote: post #6

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

Good luck getting the code. Abandon all hope if you want to make it run on a machine different from the laptop on which the original grad student implemented it.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#76

Post by kayhi »

In the world of chemistry, biochemistry and microbiology a huge step forward would be for journals to require a complete list of products used. The publication should also include the certification of analysis for each item as they vary over time.

For example, here are two product specifications for a dye called Sirius Red, the first by Sigma-Aldrich[1] and the second by Chem-Impex[2]. The Sigma-Aldrich product contains 25% dye while the Chem-Impex contains equal or greater than 21%. These two dyes could be quickly assessed with a spectrophotometer in order to determine an equivalency, however you need both dyes on hand which doesn't seems like a good use of funding. Also this touches on another problem in replication which is, what is in the other 75%+ of the bottle?

[1] http://www.sigmaaldrich.com/Graphics/COfAInfo/SigmaSAPQM/SPE... [2] http://www.chemimpex.com/MSDSDoc/22913.pdf

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#77

Post by randall »

cs702 wrote: post #21

My first reaction to this headline was "duh." Of course we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims! My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the…

I don't have anything constructive to add other than sadness. This bums me out.

Culturally academia is responsible for a lot of cult-like approaches to things. It's just a humanity problem that we have to acknowledge and use science to fight against.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#78

Post by randall »

lrem wrote: post #52
nxzero wrote: post #30

Earlier quoted context omitted.

Of course you as a reader of said claims confirm at the very least that they've been independently reproduced, right? (If so, this shouldn't be news.)

Unfortunately, there is no easy way to do it. Confirmation studies are not easily accepted by impactful journals/conferences, thus nearly nobody bothers to do them. Even if there is one, it can be surprisingly hard to find it. As a point of anecdata: my wife's master thesis was a confirmation study of using LLDA for face recognition. I remember seeing it included in some book by the university press. I gave up Googli…

There needs to be better scientific protocol. More linking through data instead of annoying cites. I think anyway.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#79

Post by return0 »

misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are.

I should think this is mostly needed in life sciences. Other, more 'exact' sciences seem to not have this problem.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#80

Post by johncolanduoni »

misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

I think the difference is that we are able to test and understand/verify the operation of the LHC at multiple levels. It's not like they just built what was on paper, flicked the ON switch, and assumed all the data that came out was correct. The components were individually tested and collisions at energies accessible by other supercolliders were done so we could compare to other results.

Unfortunately for medical drugs and psychology, researchers are mostly gathering data without an understanding of the underlying mechanisms. There are also virtually never have proposals which can be tested for compliance with reality in a quantifiable and isolated way as we can with physics, chemistry, or parts of biology.

So I feel the replication crisis is not a matter of various fields just not knowing how to do "good science", but that these fields by their nature make clean hypothesis testing vastly more difficult and p-hacking and statistical trickery (intentional or otherwise) harder to sift out.

Post reply on HN