Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

161–170 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#161

Post by xioxox »

ahmadmijot wrote: post #81
analog31 wrote: post #78

Earlier quoted context omitted.

Every experimental paper I've ever read has contained an "Experimental" section, where they provide the details on how they did it. Those sections tend to be general enough, albeit concise. In some fields, aside from specialized knowledge, good experimental work requires what we call "hands." For instance, handling air sensitive compounds, or anything in a condensed or crystalline state. In my thesis experiment, some…

> My doctoral thesis project used roughly 1/2 million dollars of gear, Wow I envy you. My doctoral thesis project spent like... USD2.5k directly for gears (half of it just to buy lego bricks to build our own instrument exactly because we can't afford to buy commercial one lol)

I used a 3 billion dollar space telescope. I don't think NASA are going to launch another to replicate some of my results.

Re: Replace peer review with “peer replication” (2021)

#162

Post by harimau777 »

ebiester wrote: post #58
nine_k wrote: post #53

Earlier quoted context omitted.

No, the idea is that the same researcher should produce k papers and n replications, instead of just k + n published papers. I'd argue that since replication is somehow faster than original research, the requirement would count a replication somewhat lower than an original paper (say, at 0.75).

That is my idea... If we opened it up, there's probably more interesting iterations, such as requiring pre-registration for all papers, having papers with pre-registration count as some portion of a full paper even if they fail so long as the pre-registration passed scrutiny, having non-replicated papers count as some portion of a fully replicated paper, and having replication as a separate category such that there i…

Is tenure really as mechanical as "publish this many papers and you get it"? My impression was that it took into account things like impact factor and was much more subjective. If that were the case, then wouldn't you run into problems with whoever decides tenure paying lip service to counting replication or failed pre-registered papers but in practice being biased in favor of original research?

Re: Replace peer review with “peer replication” (2021)

#163

Post by Nevermark »

Reproducibility would become a much higher priority if electronic versions of papers are required (by their distributors, archives, institutions, ...) to have reproduction sections, which the authors are encouraged to update over time.

UPDATABLE COVER PAGE:

Title Authors

Abstract

   Blah, blah, ...
State of reproduction:

    Not reproduced.
    Successful reproductions: ...citations...
    Reproduction attempts: ...citations...
    Countering reproductions: ...citations...
UPDATABLE REPRODUCTION SECTION ATTACHED AT END

Reproduction resources:

   Data, algorithms, processes, materials, ...
Reproduction challenges:

   Cost, time, one-off events, ...
Making this stuff more visible would help reproducers validated the value of reproduction to their home and funding institutions.

Having a standard section for this, with an initial state of "Not reproduced" provides more incentive for original workers to provide better reproduction info.

For algorithm and math work the reproduction could be served best with downloadable executable bundle.

Re: Replace peer review with “peer replication” (2021)

#164

Post by abnry »

Maxion wrote: post #149
abnry wrote: post #36

If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

Something in switzerland called the Large Hadron Collider comes to mind. I guess we should not talk about the Higgs before someone else builds a second one and replicates the papers.

Physics is generally better since they have good statistical models and can get six sigma (or whatever) results.

And replication can be done by the same party (although an independent party is better), and that may mean many trials.

And do we even set policy based on existence or non-existent of higgs bosons?

I am particularly unhappy with soft sciences in terms of replication.

Re: Replace peer review with “peer replication” (2021)

#165

Post by abnry »

Levitz wrote: post #73
abnry wrote: post #42

Earlier quoted context omitted.

If that's the case, then don't claim confidence in the work or make policy decisions based off of it. If there is no epistemological humility, then yes, it is still BS.

If any study costs X, the study and the replication costs somewhere in the ballpark of 2*X. This is not trivial.

But this is science we are talking about. A one-off lucky novel result should not be good enough. Why should our standards and our funding be so low?

Re: Replace peer review with “peer replication” (2021)

#166

Post by abnry »

fodkodrasz wrote: post #65
abnry wrote: post #36

If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

I guess if software developers will complain that it's too hard or too expensive to thoroughly test their code to ensure exactly zero bugs at release[1], then that just shows their work is BS. [1]: if you have delivered telco code to Softbank you may have heard this sentence

Replication is not the same thing as zero bugs in software.

Re: Replace peer review with “peer replication” (2021)

#167

Post by matthewdgreen »

dmarchand90 wrote: post #133
sebzim4500 wrote: post #103

Earlier quoted context omitted.

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

I believe that, contrary to popular belief, the implementation of this system would lead to a substantial increase in productivity in the long run. Here's why: Currently, a significant proportion of research results in various fields cannot be reproduced. This essentially means that a lot of work turns out to be flawed, leading to wasted efforts (you can refer to the 'reproducibility crisis' for more context). Moreov…

In the current system scientists allocate reproduction efforts to results that they intend to build on. So if you’ve claimed a breakthrough technique for levitating widgets — and I think this widget technique can be used to build spacecraft (or if I think your technique is wrong) — then I will allocate precious time and resources to reproducing your work. By contrast if I don’t think your work is significant and worth following up on, then I allocate my efforts somewhere else. The advantage is that more apparently-significant results (“might cure cancer”) tend to get a bigger slice of very limited resources, while dead-end or useless results (“might slightly reduce flatulence in cats”) don’t. This distributed entrepreneurial approach isn’t perfect, but it works better than central planning. By contrast you could adopt a Soviet-like approach where cat farts and cancer both share replication resources, but this seems like it would be bad for everyone (except the cats.)

Re: Replace peer review with “peer replication” (2021)

#168

Post by analog31 »

janalsncm wrote: post #94
analog31 wrote: post #78

Earlier quoted context omitted.

Every experimental paper I've ever read has contained an "Experimental" section, where they provide the details on how they did it. Those sections tend to be general enough, albeit concise. In some fields, aside from specialized knowledge, good experimental work requires what we call "hands." For instance, handling air sensitive compounds, or anything in a condensed or crystalline state. In my thesis experiment, some…

“Concise” isn’t good enough. If other scientists are trying to read through the tea leaves at what you’re trying to say you did, that defeats the entire point of a paper. The purpose of science is to create knowledge that other people can use and if people can’t replicate your work that’s not science.

I think the point is you don't have to give a complete BOM that includes where you got the power cables. Each scientist has to decide what amount of information needs to be conveyed. Of course this can be abused, or done sloppily, like anything else.

A place where you can spread out more is in dissertations. Mine contained an entire chapter on the experiment, another on the analysis, and appendices full of source code, schematics, etc. I happily sent out copies, at my expense. My setup was replicated roughly 3 times.

Re: Replace peer review with “peer replication” (2021)

#169

Post by jltsiren »

hedora wrote: post #137

The website dies if I try to figure out who the author (“sam”) is, but it sounds like they are used to some awful backwater of academia. They have this idea that a single editor screens papers to decide if they are uninteresting or fundamentally flawed, then they want a bunch of professors to do grunt work litigating the correctness of the experiments. In modern (post industrial revolution) branches of science, the w…

The computer science practice you describe is the exception, not the norm. It causes a lot of trouble when evaluating the merits of researchers, because most people in the academia are not familiar with it. In many places, conference papers don't even count as real publications, putting CS researchers at a disadvantage.

From my point of view, the biggest issue is accepting/rejecting papers based on first impressions. Because there is often only one round of reviews, you can't ask the authors for clarifications, and they can't try to fix the issues you have identified. Conferences tend to follow fashionable topics, and they are often narrower in scope than what they claim to be, because it's easier to evaluate papers on topics the program committee is familiar with.

The work done by the program committee was not even supposed to be proper peer review but only the first filter. Old conference papers often call themselves extended abstracts, and they don't contain all the details you would expect in the full paper. For example, a theoretical paper may omit key proofs. Once the program committee has determined that the results look interesting and plausible and the authors have presented them in a conference, the authors are supposed to write the full paper and submit it to a journal for peer review. Of course, this doesn't always happen, for a number of reasons.

Re: Replace peer review with “peer replication” (2021)

#170

Post by davidktr »

johnnyworker wrote: post #123
davidktr wrote: post #108

Earlier quoted context omitted.

You just described the majority of scientific papers. A "working set of instructions" is not really feasible in most cases. You can't include every piece of hard- and software required to replicate your own setup.

Then don't call it science, since it doesn't contribute anything to the body of human knowledge. I think it's fascinating that we can at the same time hold things like "one is none" to be true, or that you should write tests first, but with science we already got so used to a lack of discipline that we just declare it fine. It's not hard to not climb a tower you can't get down from. It's the default, actually. You st…

Imagine two scientists, Bob and Alice. Bob has spent the last 5 years examining a theory thoroughly. Now he can explain down to the last detail why the theory does not hold water, and why generations of researchers have been wrong about the issue. Unfortunately, he cannot offer an alternative, and nobody else can follow his long winded arguments anyway.

Meanwhile, Alice has spent the last 5 years making the best possible use of the flawed theory, and published a lot of original research. Sure, many of her publications are rubbish, but a few contain interesting results. Contrary to Bob, Alice can show actual results and has publications.

Who do you believe will remain in academia? And, according to public perception, will seem more like an actual scientist?

Post reply on HN