Replace peer review with “peer replication” (2021) • Hacker News Board

Re: Replace peer review with “peer replication” (2021)

#101

Post by isaacremuant » Sun, Aug 06, 2023, 6:41 PM UTC

nine_k wrote: post #57

matthewdgreen wrote: post #46

The purpose of science publications is to share new results with other scientists, so others can build on or verify the correctness of the work. There has always been an element of “receiving credit” to this, but the communication aspect is what actually matters from the perspective of maximizing scientific progress. In the distant past, publication was an informal process that mostly involved mailing around letters,…

For sharing results widely, there's arxiv. The problem is that the fanout is now overwhelming. The public perception of a publication in a prestigious journal as the established truth does not help, too.

> The public perception of a publication in a prestigious journal as the established truth does not help, too.

it's not so much the public perception but what govs/media/tech and other institutions have pushed down so that the public doesn't question whatever resulting policy they're trying to put forth.

"Trust the science" means "Thou shalt not question us, simply obey".

Anyone with eyes who has worked in institutions knows that bureocracy, careerism and corruption are intrinsic to them.

Re: Replace peer review with “peer replication” (2021)

#102

Post by johnnyworker » Sun, Aug 06, 2023, 6:43 PM UTC

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

> If someone cares enough about the work to build on it, they will replicate it anyway.

Does it really deserve to be called work if it doesn't include the a full, working set of instructions that if followed to a T allow it to be replicated? To me that's more like pollution, making it someone else's problem. I certainly don't see how "we did this, just trust us" can even be considered science, and that's not because I don't understand the scientific method, that's because I don't make a living with it, and have no incentive to not rock the boat.

Re: Replace peer review with “peer replication” (2021)

#103

Post by sebzim4500 » Sun, Aug 06, 2023, 6:45 PM UTC

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper.

I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

Re: Replace peer review with “peer replication” (2021)

#104

Post by coding123 » Sun, Aug 06, 2023, 6:45 PM UTC

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Maybe doing an experiment twice, even with a cost that is double, makes more sense so that we don't all throw away our coffee when coffee is bad, or throw away our gluten when gluten is bad, etc... (those are trivial examples) basically the cost to perform the science in many cases is so minuscule in scale to how it could affect society.

Re: Replace peer review with “peer replication” (2021)

#105

Post by hoosieree » Sun, Aug 06, 2023, 6:51 PM UTC

sebzim4500 wrote: post #103

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

Half is wildly optimistic.

Re: Replace peer review with “peer replication” (2021)

#106

Post by geysersam » Sun, Aug 06, 2023, 6:55 PM UTC

Both review and replication has their place. The mistake is treating researchers and the scientific community as a machine: "pull here, fill these forms, comment this research, have a gold star"

Let people review what they want, where they want, how they want. Let people replicate when they find interesting and motivating to work on.

Re: Replace peer review with “peer replication” (2021)

#107

Post by boxed » Sun, Aug 06, 2023, 6:57 PM UTC

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper.

I don't see how the current system works really either. Fraud is rampant, and replication crisis is the most common state of most fields.

Basically the current system is failing at finding out what is true. Which is the entire point. That's pretty damn bad.

Re: Replace peer review with “peer replication” (2021)

#108

Post by davidktr » Sun, Aug 06, 2023, 6:57 PM UTC

johnnyworker wrote: post #102

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

> If someone cares enough about the work to build on it, they will replicate it anyway. Does it really deserve to be called work if it doesn't include the a full, working set of instructions that if followed to a T allow it to be replicated? To me that's more like pollution, making it someone else's problem. I certainly don't see how "we did this, just trust us" can even be considered science, and that's not because…

You just described the majority of scientific papers. A "working set of instructions" is not really feasible in most cases. You can't include every piece of hard- and software required to replicate your own setup.

Re: Replace peer review with “peer replication” (2021)

#109

Post by tptacek » Sun, Aug 06, 2023, 7:00 PM UTC

boxed wrote: post #107

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I don't see how the current system works really either. Fraud is rampant, and replication crisis is the most common state of most fields. Basically the current system is failing at finding out what is true. Which is the entire point. That's pretty damn…

Fraud seems rampant because you hear about cases of fraud, but not about the tens of thousands of research labs plugging away day after day.

Re: Replace peer review with “peer replication” (2021)

#110

Post by User23 » Sun, Aug 06, 2023, 7:01 PM UTC

One thing that everyone needs to remember about “peer review” is that it isn’t part of the scientific method, but rather that it was imposed on the scientific enterprise by government funding authorities. It’s basically JIRA for scientists.