Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

211–220 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#211

Post by kelipso »

aftoprokrustes wrote: post #178
kergonath wrote: post #171

Earlier quoted context omitted.

Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.

Post doc? In what kind of utopian field did you work? In my former institute virtually all papers were written by PhD candidates, and reviewed by PhD candidates. With the expected effect on quality (due to lack of experience and impostor-syndrome-induced "how can I propose to reject? They are likely better than me"). But the Prof-to-postdoc-to-PhD-ratio was particularly bad (1-2-15).

I was reviewing papers starting second semester of grad school with my advisor just signing off on it, so not even PhD candidates, and it was the same for my lab mates too.

Initially we spent probably a few hours on a paper for peer review because we were relatively unfamiliar with the field but eventually I spent maybe a couple of hours doing the review. Wouldn't say peer review is a joke but it's definitely overrated by the public.

Re: Replace peer review with “peer replication” (2021)

#213

Post by geysersam »

throwaway4aday wrote: post #180
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

What's the value in publishing something that is never replicated? If no one ever reproduces the experiment and gets the same results then you don't know if any interpretations based on that experiment are valid. It would also mean that whatever practical applications could have come from the experiment are never realized. It makes the entire pursuit seem completely useless.

It still has value if we assume the experiment was done by competent honest people who are unlikely to try to fool us on purpose and unlikely do have made errors.

It would be even better if it was replicated of course.

Depending on what certainty you need you might have to wait for the result of one or several replications, but that is application dependent.

Re: Replace peer review with “peer replication” (2021)

#214

Post by pajushi »

Why shouldn't we hold science more accountable?

"Science needs accounting" is a search I had saved for months which really resonates with the idea of "peer replication."

In accounting, you always have checks and balances, you never are counting money alone. In many cases, accountants duplicate their work to make sure that it is accurate.

Auditors are the corollary to the peer review process. They're not there to redo your work, but to verify that your methods and processes are sound.

Re: Replace peer review with “peer replication” (2021)

#215

Post by geysersam »

dongping wrote: post #190
wizofaus wrote: post #182

Earlier quoted context omitted.

> What's the value in publishing something that is never replicated? Because it presents an experimental result to other scientists that they may consider worth trying to replicate?

Then those unconfirmed results are better put on arxiv, instead of being used to evaluate the performance of scientists. Tenure and grant committees should only consider replicated work.

I don't agree. A published article should not be taken for Gods Truth no matter if it's replicated or peer reviewed.

Lots of "replicated" "peer-reviewed" research have been found to be wrong. That's fine, it's part of the process of discovery.

A paper should be taken for what it is: a piece of scientific work, a part of a puzzle.

Re: Replace peer review with “peer replication” (2021)

#216

Post by tnecniv »

p1esk wrote: post #118
justinpombrio wrote: post #52

Earlier quoted context omitted.

> If someone cares enough about the work to build on it, they will replicate it anyway. Well, the trouble is that hasn't been the case in practice. A lot of the replication crisis was attempting for the first time to replicate a foundational paper that dozens of other papers took as true and built on top of, and then seeing said foundational paper fail to replicate. The incentives point toward doing new research inst…

It is the case in my field (ML): if I care enough about a published result I try to replicate it.

This is something very sensible in ML since, you likely want to use that algorithm for something else (or to extend / modify it), so you need to get it working in your pipeline and verify it works by comparing with the published result.

In something like psychology that is likely harder, since the experiment you want to do might be related to but differ significantly from the prior work. I am no psychologist, but I’d like to think that they don’t take one study as ground truth for that reason but try to understand causal mechanisms with multiple studies as data points. If the hypothesis is correct, it will likely present in multiple ways.

Re: Replace peer review with “peer replication” (2021)

#217

Post by RoyalHenOil »

omgwtfbyobbq wrote: post #184
p1esk wrote: post #124

Earlier quoted context omitted.

Who will be replicating it? Why would I want to set aside my own research to replicate some claim someone made? How would this help my career?

I dunno. Offhand, I guess whoever is citing the work would need to replicate it, but only if it's cited sufficiently (overall number of citations, considered foundational, etc...) This could help your career by increasing the probability that the work you're citing is more likely accurate, and as a result, your work is also likely more accurate.

A typical paper may cite dozens or hundreds of other papers. This does not sound feasible. It honestly seems like it would worsen the existing problem and force scientists to focus even more on their own original research in isolation from others, to avoid the expense of running myriad replication experiments that they likely don't have the funding and personnel to do.

Re: Replace peer review with “peer replication” (2021)

#218

Post by pvaldes »

coding123 wrote: post #104
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Maybe doing an experiment twice, even with a cost that is double, makes more sense so that we don't all throw away our coffee when coffee is bad, or throw away our gluten when gluten is bad, etc... (those are trivial examples) basically the cost to perform the science in many cases is so minuscule in scale to how it could affect society.

One. Doing experiments is yet enough difficult and painful.

Two. This drain of resources can't be done for free. Somebody will need to pay twice for half of the research [1], and faster. Peers will need to be hired and paid, maybe by the writer's grants. Researchers cant justify to give their own funds to other teams without a profound change in regulation and even in that case would be harming their own projects.

[1] as the valuable experts are now stuck validating things instead doing their own job

Would open also a door for foul play. Blocking competitors teams in molasses just trowing them secondary silly problems that they know that are a dead end, while the other team work in the real deal, and take the advantage to win the patent.

Re: Replace peer review with “peer replication” (2021)

#219

Post by SonOfLilit »

My first thought was "this would never work, there is so much science being published and not enough resources to replicate it all".

Then I remembered that my main issue with modern academia is that everyone is incentivized to publish a huge amount of research that nobody cares about, and how I wish we would put much more work into each of much fewer research directions.

Re: Replace peer review with “peer replication” (2021)

#220

Post by johnnyworker »

varjag wrote: post #208
johnnyworker wrote: post #187

Earlier quoted context omitted.

> But the original result is useful in its own right - it tells us that there may be something worth exploring. I disagree. It shows that when someone writes something in a text editor and publishes it, others can read the words they wrote. That's all it shows, by itself. Just like someone writing something on the web only tells us that a textarea accepts just about any input. And even if it did show more than that,…

> Note I didn't say your machine, I said one machine of yours. This thread discusses peer replication, this is not even an analogy.

If you can't even replicate it yourself, what makes you think peers could? We are talking about something not being replicated, not even by the original author. The most extreme version would be something that you could only get to run once on the same machine, and never on any other machine.
Post reply on HN