Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

311–320 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#311

Post by lou1306 »

mike_hearn wrote: post #204
titzer wrote: post #86

In the PL field, conferences have started to allow authors to submit packaged artifacts (typically, source code, input data, training data, etc) that are evaluated separately, typically post-review. The artifacts are evaluated by a separate committee, usually graduate students. As usual, everything is volunteer. Even with explicit instructions, it is hard enough to even get the same code to run in a different environ…

Is PL theory actually science? Although we call it computer science, I don't personally think CS is actually a science in the sense of studying nature to understand it. Computers are artificial constructs. CS is a lot closer to engineering than science. Indeed it's kind of nonsensical to talk about replicating an experiment in programming language theory. For the "hard" sciences, replication often isn't so difficult…

Well there are entire areas of CS research tangential to PLs (say, SAT/SMT, software verification, program synthesis) where the fundamental problems are known to be NP-complete/exponential/undecidable. So it is pretty hard to declare a new approach "superior" on purely theoretical grounds: you usually have to run some benchmarks and see how your new approach compares to existing alternatives. And you want these benchmark to be replicable across different machines and platforms.

Re: Replace peer review with “peer replication” (2021)

#312

Post by philwelch »

crote wrote: post #305
sebzim4500 wrote: post #103

Earlier quoted context omitted.

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

Where are you going to get the budget to build a second LHC solely for replication ? How are you going to replicate a long-term medical cohort study which has been running for thirty years? What about a paper describing a one-off astronomical event, like the "Wow!" signal? What if you research the long-term impact of high-dose radiation exposure during Chernobyl? There is plenty of science out there which financially…

I think your examples are more amenable to replication than you assume:

> Where are you going to get the budget to build a second LHC solely for replication?

In cases like this you could simply have a second, independent team time-sharing the LHC and using it to replicate experiments run by the first team. (And vice- versa). It’s not a perfect replication but it’s probably still an improvement over the “just trust me bro” status quo.

> How are you going to replicate a long-term medical cohort study which has been running for thirty years?

Run two independent studies in parallel from the beginning.

> What about a paper describing a one-off astronomical event, like the "Wow!" signal?

There was a ton of effort invested into trying to replicate that observation! Since nobody else ever managed to do so, we can’t draw any conclusions from it.

> What if you research the long-term impact of high-dose radiation exposure during Chernobyl?

That doesn’t preclude replication unless, for some reason, you’re the only researcher researching the long-term impact of high-dose radiation exposure during Chernobyl.

Re: Replace peer review with “peer replication” (2021)

#313

Post by jakear »

optionalsquid wrote: post #292
jakear wrote: post #258

Earlier quoted context omitted.

Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.

You frequently see this kind of reasoning in medical quackery: > Butt-candling[1] must work, just look all these happy customers! But history is replete with ineffective or downright harmful treatments being popular long after the evidence showed them to be ineffective or harmful. Homeopathy is a prime example of this, seeing as those concoctions contain either no active ingredients or (in cases of low dilutions stil…

This is a perfect example. In the real world, some placebos work. You take them, believe that they will work, and they do. Fantastic! Your problem has been solved, you share your story with friend(s), the cure spreads based on how well it works for them.

It's only in academia that they must instead work in a sterile environment that some person who has never stepped foot off a school campus thinks is "more authentic" for them to be seen as legitimate.

Re: Replace peer review with “peer replication” (2021)

#314

Post by jakear »

depereo wrote: post #273
jakear wrote: post #258

Earlier quoted context omitted.

Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.

This doesn't really account for fraud or abuse. Plenty of profitable companies whose product is a scam, or has other factors that make it successful. Market success isn't the same thing as validity.

Peer review isn't the same thing as validity either. Both are proxies, sales are simply a much better proxy. Especially long term sales from an established company.

Look at it like this: a Peer Reviewed (tm) article comes out saying "Foos cannot Bar, it is impossible". The same day, Apple releases "Bar for Foos, $8/month". Over the next year you see media outlets discussing how well Foo Barring works. Online reviewers talk about how they've incorporated Barring their Foos into everyday life and it has benefitted them in all these great ways. Your colleagues at work mention their fruitful Foo Barring adventure over the past weekend. Routine posts on HN come up where hackers describe how they've incorporated the Foo Barring API's into their own products in some novel way.

Your mother then calls you up to ask if she should get involved with this new Foo Barring thing. What do you say, can Foos Bar?

Re: Replace peer review with “peer replication” (2021)

#315

Post by amai »

It would already be a step in the right direction, if papers would also publish a VM with all their code, data and dependencies. It is nice to have the code (https://blog.arxiv.org/2020/10/08/new-arxivlabs-feature-prov...), but without necessary dependencies, the correct OS, compiler Version, etc. replication is even with code often impossible.

Having running demos is another step in the right direction (see https://blog.arxiv.org/2022/11/17/discover-state-of-the-art-...).

But outside of computer science replication is even more difficult. Maybe if people would use standardized laboratories and robots, one could replicate findings by rerunning the robots code on another standard robot lab ( Basically the idea here is to virtualize laboratory work).

But even then for the biggest most complex experiments this will not work: Replicate CERN anyone?

Re: Replace peer review with “peer replication” (2021)

#316

Post by kergonath »

kibibu wrote: post #270
jakear wrote: post #252

Earlier quoted context omitted.

Industry doesn't need peer review. The proof of the pudding is in the sales.

Then why does it dress its marketing up in academic paper format?

It gives a nice patina and some legitimacy. They have no need to publish quasi-scientific articles.

Re: Replace peer review with “peer replication” (2021)

#317

Post by kergonath »

godelski wrote: post #302
kergonath wrote: post #171

Earlier quoted context omitted.

Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.

> with unpaid reviewers who have many other, more interesting things to do It's kinda funny. A journal doesn't make the product it sells (the papers that it copywrites). It doesn't pay for the service it performs ("vetting" and editing). And both of these would be done regardless of their existence. I can understand distribution, but that hasn't been useful for over a decade now. What even do these things do anymore?…

> What even do these things do anymore?

Networking, mostly, in the sense that an article in a high impact journal has a higher probability to be integrated in citations networks. The fact that there is some gate keeping means that it’s valuable to be in rather than out, and that’s something you can use to get a position. Also, better journals (which are not necessarily the highest-impact ones) tend to have more thorough peer review (such as 3 reviewers by default instead of 1 or 2, editors who are not afraid to ask for more reviews if the 3 are not conclusive, etc).

> (btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one)

I am lucky not to have been in that situation when I was a student, and I did not delegate any further when I got the occasional review from the prof when I was a post-doc. But I am unfortunately not surprised.

Re: Replace peer review with “peer replication” (2021)

#318

Post by whatever1 »

We can have tiers. Tier 1 peer reviewed. Tier 2 peer replicated. We can have it as a stamp on the papers.

All PhD programs have requirement for a minimum number of novel publications. We could add to the requirements a minimum number of replications.

But truth to be told, a PhD in science/ engineering will probably spend their first two years trying to replicate the SOTA anyway. It’s just that today you cannot publish this effort, nobody cares, except yourself and your advisor.

Re: Replace peer review with “peer replication” (2021)

#319

Post by bartwr »

I review 10-20 papers a year.

It's a ton of unpaid, volunteer work, if I want to be a high quality reviewer then it's at least a day (at least 3 thorough reads, taking notes, writing the review, reviewer discussions, post rebuttal, back-and-forth for journals). I am lucky and privileged that my employer counts this towards work time. Only 20% papers get accepted in my domain.

Now if I had to spend a week on replicating a paper - and this is CS/graphics, where it's easy and "free" - I'd never volunteer to being a reviewer.

You'd need professional "replicators", but who will pay for them? And who will be them - you need experts, and if you are an expert, you don't want to merely replicate others people work full time, instead of working on your own innovation.

Re: Replace peer review with “peer replication” (2021)

#320

Post by picadores »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Yes, the amount of work done that could go into more paper churn?
Post reply on HN