Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

301–310 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#301

Post by godelski »

vibrio wrote: post #156
kergonath wrote: post #50

Earlier quoted context omitted.

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort…

“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.

It's because we all know it is a con to just get money from universities and governments. We had a long time without this system and we perverted it from its original intent. 3-4 people reading a paper are incapable of proving something is valid (the inverse is true, but also remember there's 3 options: true, false, indeterminate). It's even more silly that we think we can rank them (i.e. top tier venues). Rejection rate isn't a ranking, and it is silly to think it could be (way too easy to hack). We just go with the momentum and because that's what the admins require of us. Otherwise, let's me honest, we're all communicating our work outside the structure of journals and conferences (at least they offer meetups). It's very hard for me to justify their existence if we're being honest. And honestly, I don't see how abandoning these systems wouldn't just save us time and money. Like god, the number of hours I spend reworking things which got no actionable feedback from the previous round, is insane. It is just too easy to get non-actionable rejections and that just wastes everybody's time and money, and frankly holds back science.

Re: Replace peer review with “peer replication” (2021)

#302

Post by godelski »

kergonath wrote: post #171
vibrio wrote: post #156

Earlier quoted context omitted.

“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.

Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.

> with unpaid reviewers who have many other, more interesting things to do

It's kinda funny. A journal doesn't make the product it sells (the papers that it copywrites). It doesn't pay for the service it performs ("vetting" and editing). And both of these would be done regardless of their existence. I can understand distribution, but that hasn't been useful for over a decade now. What even do these things do anymore?

(btw, I've seen profs delegate to undergrads. And it is quite common for post-docs AND grad students to be reviewers. Trust me, I am one)

Re: Replace peer review with “peer replication” (2021)

#303

Post by godelski »

aftoprokrustes wrote: post #178
kergonath wrote: post #171

Earlier quoted context omitted.

Yes, because we know how the metaphorical sausage is made: with unpaid reviewers who have many other, more interesting things to do and often an axe to grind. That is, if they don’t delegate the review to one of their post-docs.

Post doc? In what kind of utopian field did you work? In my former institute virtually all papers were written by PhD candidates, and reviewed by PhD candidates. With the expected effect on quality (due to lack of experience and impostor-syndrome-induced "how can I propose to reject? They are likely better than me"). But the Prof-to-postdoc-to-PhD-ratio was particularly bad (1-2-15).

> "how can I propose to reject? They are likely better than me"

Funny enough, I see exactly the opposite. I've seen this in both reviews I've done and reviews I've received. Just this week I reviewed and saw one of my fellow reviewers write in their justifications: I am not familiar with X, but I am skeptical that the method can scale to a more complex application. Their weaknesses section was extremely generic and it was very clear they didn't understand the work. They gave a weak reject. In fact, when I first started reviewing, I was explicitly told to _only_ accept if I was confident that the work was good. So in my experience, the bias goes the other way that you are proposing.

Btw, I've even seen undergrads acting as reviewers. I was asked to review in in my first year of grad school. I don't think I was qualified then, but I was always a junior reviewer rather than a full so idk.

Re: Replace peer review with “peer replication” (2021)

#304

Post by jltsiren »

mike_hearn wrote: post #294
matthewdgreen wrote: post #232

Earlier quoted context omitted.

The wonderful thing about the western world is that most countries value freedom of the press. The dark side of this is that you can spin up your own “scientific journal” and charge people to publish in it, game the rankings like any common SEO scam, and nobody will stop you because (especially here in the US) you’re exercising your first amendment rights. Then people can fill it with nonsense and even script-generat…

With respect, I think you should research the topic more deeply before assuming that this is some sort of fringe problem that doesn't exist in the "actual" scientific community. The second link I provided is by Nature News and states specifically that the problem affects "prestigious journals" (their words). Auto-generated papers have been published in journals from the IEEE, Elsevier, Springer Nature and other well…

As far as research is concerned, the names you mentioned are not prestigious brands. Universities are organizations that provide facilities and services in exchange for their share of grant money. Publishers publish whatever people are willing to pay for. Prominent mentions of supposedly prestigious institutions are a red flag in science reporting. Very often, either the writer is trying to promote something, or they don't know what they are talking about.

There are two parallel academias. There is the reputable high-trust one, where it's easy to get away with fraud, because people generally don't commit it. And there is the scammy one that exists to help people to game the metrics. While the two overlap a bit, they are mostly disjoint.

If you are an academic, you get a steady stream of spam from the scammy side of the academia. You get calls to submit papers to a conference with "Proceedings by Springer" (but the scope of the conference is barely mentioned), you get invited to become an "ΕԀitоrial ΜҽmƄҽr" of a journal, and so on. Those are like Nigerian letters. They make it very explicit that they are scams, in order to avoid wasting people's time.

You guessed that nobody reads the journal you mentioned, and that's trivially true. Of course nobody reads journals, because their scopes are too wide. No matter what you are working on, most articles in the journals you publish in are irrelevant to you. People read only articles that look interesting or relevant. If nobody cares about an article, it doesn't get read.

While the rest of the world is based on top-down hierarchies, that's not a good model for understanding research. In general, the higher up in the hierarchy you go, the less relevant things become. The article is more relevant than the journal, and the journal is more relevant than the publisher. A rank-and-file professor is more relevant than a department chair. A department chair is more relevant than a dean. And a dean is more relevant than a chancellor/president/whatever.

Re: Replace peer review with “peer replication” (2021)

#305

Post by crote »

sebzim4500 wrote: post #103
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

Where are you going to get the budget to build a second LHC solely for replication? How are you going to replicate a long-term medical cohort study which has been running for thirty years? What about a paper describing a one-off astronomical event, like the "Wow!" signal? What if you research the long-term impact of high-dose radiation exposure during Chernobyl?

There is plenty of science out there which financially, practically, or ethically simply by definition cannot be replicated. That doesn't mean their results should not be published. If peer review shows that their methods and analysis are sound, there is no reason to doubt the results.

Re: Replace peer review with “peer replication” (2021)

#307

Post by ljf »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Indeed, a friend of my was running experiments on quantum computing - the first 18 months on each run of his tests, as he moved between jobs and universities, was just setting up the necessary refrigeration systems (and generally building them from scratch each time).

Only then could he even start building the experiment - total time to run it all seems to run across years.

Re: Replace peer review with “peer replication” (2021)

#308

Post by JR1427 »

One thing I think people are missing, is that labs replicate other experiments all the time as part of doing their own research. It's just that the results are not always published, or not published in a like-for-like way.

But the information gets around. In my former field, everyone knew which were the dodgy papers, with results no-one could replicate.

Re: Replace peer review with “peer replication” (2021)

#309

Post by rkangel »

dmarchand90 wrote: post #133
sebzim4500 wrote: post #103

Earlier quoted context omitted.

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

I believe that, contrary to popular belief, the implementation of this system would lead to a substantial increase in productivity in the long run. Here's why: Currently, a significant proportion of research results in various fields cannot be reproduced. This essentially means that a lot of work turns out to be flawed, leading to wasted efforts (you can refer to the 'reproducibility crisis' for more context). Moreov…

It is analogous to spending half your time writing tests when developing software. Yes it slows you down in the very short term, but in the medium and long term it speeds you up as you have confidence in the code and there aren't bugs lurking everywhere.

Re: Replace peer review with “peer replication” (2021)

#310

Post by m-watson »

JR1427 wrote: post #308

One thing I think people are missing, is that labs replicate other experiments all the time as part of doing their own research. It's just that the results are not always published, or not published in a like-for-like way. But the information gets around. In my former field, everyone knew which were the dodgy papers, with results no-one could replicate.

That is something that I have struggled to convey. Working in a scientific field looks a lot different than just reading things from that scientific community. Sub-fields and up real small and you know what is going on, what the problems are, and who are what kind of players in your field.
Post reply on HN