Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

141–150 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#141

Post by magimas »

orangepurple wrote: post #128
advisedwang wrote: post #115

Earlier quoted context omitted.

It would be more than just half productivity. Not only do you have to do the work twice, but you add the delay of someone else replicating before something can be published and built upon by others. If you are developer, imagine how much your productivity would drop going from a 3 minute build to a 1 day build.

Terrible analogy. It might take months to come up with an idea but another should be able to follow your method and implement it much more quickly than it took you to come up with the concept and implement it.

horrible take. Taking the LK99 situation as an example: simply copying and adapting a well described growth recipee to your own setup and lab conditions may take weeks. And how would you address situations where measurement setups only exist once on the earth? How would you do peer replication of LHC measurements? Wait for 50 years till the next super-collider is built and someone else can finally verify the results? On a smaller scale: If you need measurements at a synchrotron radiation source to replicate a measurement, is someone supposed to give up his precious measurement time to replicate a paper he isn't interested in? And is the original author of a paper that's in the queue for peer replication supposed to wait for a year or two till the reviewer gets a beamtime on an appropriate measurement station? Even smaller: I did my PhD in a lab with a specific setup that only a single other group in the world had an equivalent to. You simply would not be able to replicate these results.

Peer replication is completely unfeasible in experimental fields of science. The current process of peer review is alright, people just need to learn that single papers standing by themselves don't mean too much. The "peer replication" happens over time anyway when others use the same tools, samples, techniques on related problems and find results in agreement with earlier papers.

Re: Replace peer review with “peer replication” (2021)

#142

Post by Maxion »

sebastos wrote: post #100
matthewdgreen wrote: post #46

The purpose of science publications is to share new results with other scientists, so others can build on or verify the correctness of the work. There has always been an element of “receiving credit” to this, but the communication aspect is what actually matters from the perspective of maximizing scientific progress. In the distant past, publication was an informal process that mostly involved mailing around letters,…

Very well put. This is the clearest way of looking at it in my view. I’ll pile on to say that you also have the variable of how the non-scientist public gleans information from the academics. Academia used to be a more insular cadre of people seeking knowledge for its own sake, so this was less relevant. What’s new here is that our society has fixated on the idea that matters of state and administration should be sig…

A significant problem we face today is the way research, especially in academia, gets spotlighted in the media. They often hyper-focus on single studies, which can give a skewed representation of scientific progress.

The reality is that science isn't about isolated findings; it's a cumulative effort. One paper might suggest a conclusion, but it's the collective weight of multiple studies that provides a more rounded understanding. Media's tendency to cherry-pick results often distorts this nuanced process.

It's also worth noting the trend of prioritizing certain studies, like large RCTs or systematic reviews, while overlooking smaller ones, especially pilot studies. Pilot studies are foundational—they often act as the preliminary research needed before larger studies can even be considered or funded. By sidelining or dismissing these smaller, exploratory studies, we risk undermining the very foundation that bigger, more definitive research efforts are built on. If we consistently ignore or undervalue pilot studies, the bigger and often more impactful studies may never even see the light of day.

Re: Replace peer review with “peer replication” (2021)

#143

Post by Maxion »

casualscience wrote: post #111
matthewdgreen wrote: post #46

The purpose of science publications is to share new results with other scientists, so others can build on or verify the correctness of the work. There has always been an element of “receiving credit” to this, but the communication aspect is what actually matters from the perspective of maximizing scientific progress. In the distant past, publication was an informal process that mostly involved mailing around letters,…

Most of this is very legit, but this > Still other outsiders, who misunderstand the entire process, are upset that intermediate results are sometimes incorrect. This confuses them, and they’re angry that the process sometimes assigns “points” to people who they perceive as undeserving. So instead of simply accepting that sharing results widely to maximize the chance of verification is the whole point of the publicati…

And on the extreme end you have the multi-decade longitudinal studies in epidemiology / biomedicine that would be more-or-less impossible to replicate.

Re: Replace peer review with “peer replication” (2021)

#144

Post by Maxion »

kergonath wrote: post #29
NalNezumi wrote: post #8

Imo, A more realistic thing to do is "replicability review" and/or requirement to submit "methodology map" to each paper. The former would be a back and forth between a reviewer that inquire and ask questions (based on the paper) with the goal to reproduce the result , but don't have to actually reproduce it. This is usually good to find out missing details in the paper that the writer just took for granted everyone…

> I've met Bio PHD that have wasted Months of their life tracking up experimental details not mentioned in a paper Same. Now, when I review manuscripts, I pay much more attention to whether there is enough information to replicate the experiment or simulation. We can put out a paper with wrong interpretations and that’s fine because other people will realise that when doing their own work. We cannot let papers get pu…

> Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use.

This may be possible in some sciences, but not in epidemiology or biomed. Often the study is based on tissue samples owned by some entity, with permission granted only to some certain entity.

Datasets in epidemiology are often full of PII, and cannot be shared publicly for many reasons.

Re: Replace peer review with “peer replication” (2021)

#145

Post by simlan »

janalsncm wrote: post #91

For a while Reddit had the mantra “pics or it didn’t happen”. At least in CS/ML there needs to be a “code or it didn’t happen”. Why? Papers are ambiguous. Even if they have mathematical formulas, not all components are defined. Peer replication in these fields is an easy low hanging fruit that could set an example for other fields of science.

That is too simplistic. You underestimate the depth of academia. Sure the latest break through Alzheimers study or related research would benefit from a replication. Which is done out of commercial interest anyway.

But your run of the mill niche topic will not have the dollars behind it to replicate everyones research.just because CS/AI research is very convenient to replicate does not mean this can be extended to all research being done.

That is exactly why peer review exists to weed out the implausible and low effort/relevance work. It is not fraud proof because it was not designed to be.

Re: Replace peer review with “peer replication” (2021)

#146

Post by Maxion »

titzer wrote: post #86

In the PL field, conferences have started to allow authors to submit packaged artifacts (typically, source code, input data, training data, etc) that are evaluated separately, typically post-review. The artifacts are evaluated by a separate committee, usually graduate students. As usual, everything is volunteer. Even with explicit instructions, it is hard enough to even get the same code to run in a different environ…

> I get the impression that suggestions like these are written by non-scientists who do not have experience with the peer review process of any discipline. Things just don't work like that.

Not to mention that the cutting edge in many sciences are perhaps two-three research groups of 5-30 individuals each in varying research institutions around the world.

Re: Replace peer review with “peer replication” (2021)

#147

Post by TrackerFF »

Seems to have been hugged to death.

But - a quick counterexample - as far as replication goes: What if the experiments were run on custom made or exceedingly expensive equipment? How are the replicators supposed to access that equipment? Even in fields which are "easy" to replicate - like machine learning - we are seeing barriers of entry due to expensive computing power. Or data collection. Or both.

But then you move over to physics, and suddenly you're also dealing with these one-off custom setups, doing experiments which could be close to impossible to replicate (say you want to conduct experiments on some physical event that only occurs every xxxx years or whatever)

Re: Replace peer review with “peer replication” (2021)

#148

Post by sqrt_1 »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

FYI there is a at least one science journal that only publishes reproduced research:

Organic Syntheses "A unique feature of the review process is that all of the data and experiments reported in an article must be successfully repeated in the laboratory of a member of the editorial board as a check for reproducibility prior to publication"

https://en.wikipedia.org/wiki/Organic_Syntheses

Re: Replace peer review with “peer replication” (2021)

#149

Post by Maxion »

abnry wrote: post #36

If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

Something in switzerland called the Large Hadron Collider comes to mind.

I guess we should not talk about the Higgs before someone else builds a second one and replicates the papers.

Re: Replace peer review with “peer replication” (2021)

#150

Post by MrJohz »

johnnyworker wrote: post #134
MrJohz wrote: post #127

Earlier quoted context omitted.

I work with code, which is about as reproducible as it is possible to get - the artifacts I produce are literally just instructions on how to reproduce the work I've done again, and again, and again. And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine, despite the two environments being as identical as I can possibly make them. I agree that reprod…

> And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine But this is the other way around. Have you ever written a program that doesn't run anywhere except a single machine of yours? Would you release it and advertise it and encourage other people to use it as dependency in their software? If it only runs on one machine of yours, you don't even know…

I think you misunderstand the point of scientific publication here (at least in theory, perhaps less so in practice). The purpose of a paper is typically to say "I have achieved these results in this environment (as far as I can tell)", and encourages reproduction. But the original result is useful in its own right - it tells us that there may be something worth exploring. Yes, it may just be a measurement error (I remember the magic faster than light neutrinos), but if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly.

And yes, there are often overly excited press releases that accompany it - the "advertise it and encourage others to us it as a dependency" part of it analogy - but this is typically just noise in the context of scientific research. If that is your main problem with scientific publishing, you may want to be more critical of science journalism instead.

Fwiw, yes of course I've written code that only runs on my machine. I imagine everyone has, typically accidentally. You do it, you realise your mistake, you learn something from it. Which is exactly what we expect from scientific papers that can't be reproduced.

Post reply on HN