Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

51–60 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#51

Post by kshahkshah »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

When I looked into this, more than 15 years ago, I thought the difficult portion wasn't sharing the recipe, but the ingredients, if you will - granted I was in a molecular biology lab. Effectively the Material Transfer Agreements between Universities all trying to protect their IP made working with each other unbelievably inefficient.

You'd have no idea if you were going down a well trodden path which would yield no success because you have no idea it was well trod. No one publishes negative results, etc.

Re: Replace peer review with “peer replication” (2021)

#52

Post by justinpombrio »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

> If someone cares enough about the work to build on it, they will replicate it anyway.

Well, the trouble is that hasn't been the case in practice. A lot of the replication crisis was attempting for the first time to replicate a foundational paper that dozens of other papers took as true and built on top of, and then seeing said foundational paper fail to replicate. The incentives point toward doing new research instead of replication, and that needs to change.

Re: Replace peer review with “peer replication” (2021)

#53

Post by nine_k »

SkyMarshal wrote: post #49
ebiester wrote: post #30

Earlier quoted context omitted.

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomizat…

> x fewer papers but x number of replications, and you are expected to have x replications in your specialty. Could it be simplified it even further to say x number of papers, but they only count if they’re replicated by others in the field?

No, the idea is that the same researcher should produce k papers and n replications, instead of just k + n published papers.

I'd argue that since replication is somehow faster than original research, the requirement would count a replication somewhat lower than an original paper (say, at 0.75).

Re: Replace peer review with “peer replication” (2021)

#54

Post by dmbche »

matthewdgreen wrote: post #46

The purpose of science publications is to share new results with other scientists, so others can build on or verify the correctness of the work. There has always been an element of “receiving credit” to this, but the communication aspect is what actually matters from the perspective of maximizing scientific progress. In the distant past, publication was an informal process that mostly involved mailing around letters,…

Your analysis seems to portray all scientists as pure hearted. May I remind you of the latest Stanford scandal where the president of Stanford was found to have manipulated data?

Today, publications do not serve the same purpose as they did before the internet. It is trivial today to write a convincing paper without research and getting that published(www.theatlantic.com/ideas/archive/2018/10/new-sokal-hoax/572212/&sa=U&ved=2ahUKEwjnp5mRtsiAAxVwF1kFHesBDC8QFnoECAkQAg&usg=AOvVaw0t_Bo31BrT5D9zHBdmNAqi).

Re: Replace peer review with “peer replication” (2021)

#55

Post by ebiester »

Eddy_Viscosity2 wrote: post #37
ebiester wrote: post #30

Earlier quoted context omitted.

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomizat…

It's not easy because it isn't simple. How do get all of the universities to change their incentives to back this?

We agree - the "simple not easy" turn of phrase is speaking to that point. It is easy once implemented, but it isn't easy to transition. (I am academia-adjacent by marriage but closer to the humanities, so I understand the amount of work it would take to perform the transition.)

Re: Replace peer review with “peer replication” (2021)

#57

Post by nine_k »

matthewdgreen wrote: post #46

The purpose of science publications is to share new results with other scientists, so others can build on or verify the correctness of the work. There has always been an element of “receiving credit” to this, but the communication aspect is what actually matters from the perspective of maximizing scientific progress. In the distant past, publication was an informal process that mostly involved mailing around letters,…

For sharing results widely, there's arxiv. The problem is that the fanout is now overwhelming.

The public perception of a publication in a prestigious journal as the established truth does not help, too.

Re: Replace peer review with “peer replication” (2021)

#58

Post by ebiester »

nine_k wrote: post #53
SkyMarshal wrote: post #49

Earlier quoted context omitted.

> x fewer papers but x number of replications, and you are expected to have x replications in your specialty. Could it be simplified it even further to say x number of papers, but they only count if they’re replicated by others in the field?

No, the idea is that the same researcher should produce k papers and n replications, instead of just k + n published papers. I'd argue that since replication is somehow faster than original research, the requirement would count a replication somewhat lower than an original paper (say, at 0.75).

That is my idea... If we opened it up, there's probably more interesting iterations, such as requiring pre-registration for all papers, having papers with pre-registration count as some portion of a full paper even if they fail so long as the pre-registration passed scrutiny, having non-replicated papers count as some portion of a fully replicated paper, and having replication as a separate category such that there is a minimum k, a minimum n, and a minimum k+n.

The non-easy part of this is once we start making changes to the criteria for tenure, this opens up people trying to stuff all the solutions for all of the problems that everyone knows already. (See Above.) Would some one try to stuff code-available for CS conference papers, for example? What does it mean for a poster session? At what point are papers released for pre-print? What does it mean for the tenure clock or the Ph.D clock? Does it mean that pre-tenure can't depend on studies that take time to replicate? What do we do with longitudinal studies?

I think you're looking at a 50 year transition where you would have to start simple and iterate.

Re: Replace peer review with “peer replication” (2021)

#59

Post by msla »

eesmith wrote: post #27
msla wrote: post #5

Earlier quoted context omitted.

With some of the things, but admittedly not most of the things you mentioned, there's a dataset (somewhere) and some code run on that dataset (somewhere) and replication would mean someone else being able to run that code on that dataset and get the same results. Would this require labs to improve their software environments and learn some new tools? Would this require labs to give up whatever used to be secret sauce…

> someone else being able to run that code on that dataset and get the same results. I think when people talk about "replicate" they mean something more than that. The dataset could contain coding errors, and the analysis could contain incorrect formulas and bad modeling. Reproducing a bad analysis, successfully, provide no corrective feedback. I know for one paper I could replicate the paper's results using the pape…

My point is that we can, apparently, improve the baseline expectations in the parts of science where this kind of reproducibility is possible. That isn't all science, granted, but it is some science. It isn't a panacea, granted, but it could guard against some forms of misconduct or honest error some of the time. The self-correcting part of science only works when there's something for it to work on, so open data and runnable code ought to improve that self-correction mechanism.

Re: Replace peer review with “peer replication” (2021)

#60

Post by matthewdgreen »

dmbche wrote: post #54
matthewdgreen wrote: post #46

The purpose of science publications is to share new results with other scientists, so others can build on or verify the correctness of the work. There has always been an element of “receiving credit” to this, but the communication aspect is what actually matters from the perspective of maximizing scientific progress. In the distant past, publication was an informal process that mostly involved mailing around letters,…

Your analysis seems to portray all scientists as pure hearted. May I remind you of the latest Stanford scandal where the president of Stanford was found to have manipulated data? Today, publications do not serve the same purpose as they did before the internet. It is trivial today to write a convincing paper without research and getting that published(www.theatlantic.com/ideas/archive/2018/10/new-sokal-hoax/572212/&s…

No subset of humanity is “pure hearted.” Fraud and malice will exist in everything people do. Fortunately these fraudulent incidents seem relatively rare, when one compares the number of reported incidents to the number of publications and scientists. But this doesn’t change anything. The benefit of scientific publication is to make it easier to detect and verify incorrect results, which is exactly what happened in this case.

I understand that it’s frustrating it didn’t happen instantly. And I also understand that it’s deeply frustrating that some undeserving person accumulated status points with non-scientists based on fraud, and that let them take a high-status position outside of their field. (I think maybe you should assign some blame to the Stanford Trustees for this, but that’s up to you.) None of this means we’d be better off making publication more difficult: it means the metrics are bad.

PS When a TFA raises something like “the replication crisis” and then entangles it with accusations of deliberate fraud (high profile but exceedingly rare) it’s like trying to have a serious conversation about automobile accidents, but spending half the conversation on a handful of rare incidents of intentional vehicular homicide. You’re not going to get useful solutions out of this conversation, because it’s (perhaps deliberately) misunderstanding the impact and causes of the problem.

Post reply on HN