Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

241–250 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#241

Post by wilde »

advisedwang wrote: post #115
sebzim4500 wrote: post #103

Earlier quoted context omitted.

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

It would be more than just half productivity. Not only do you have to do the work twice, but you add the delay of someone else replicating before something can be published and built upon by others. If you are developer, imagine how much your productivity would drop going from a 3 minute build to a 1 day build.

You can easily look up non peer reviewed papers on arxiv. Why would this be different with replication?

Re: Replace peer review with “peer replication” (2021)

#242

Post by wilde »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

The pressure to replicate would make folks publish things in forms that are easier to replicate. This cost would go down over time.

Re: Replace peer review with “peer replication” (2021)

#243

Post by PaulHoule »

jakear wrote: post #172
vibrio wrote: post #156

Earlier quoted context omitted.

“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.

It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".

Except for the ones that conclude it is all a scam.

Re: Replace peer review with “peer replication” (2021)

#244

Post by PaulHoule »

tnecniv wrote: post #200
evandrofisico wrote: post #173

Earlier quoted context omitted.

Usually coming up with a idea is the easy part. For example, in my PhD project, i started with an idea from my advisor that he had in the early 2000. Implementing the code for the simulation and analysis of the data? four months, at most. Running the simulation? almost three years until I had data with good enough resolution for publishing.

It’s also very easy to come up with bad ideas — I did plenty of that and I still do, albeit less than I used to. Finding an idea that is novel, interesting, and tractable given your time, skills, resources, and knowledge of the literature is hard, and maybe the most important skill you develop as a researcher. For a reductive example, the idea to solve P vs NP is a great one, but I’m not going to do that any time soo…

I'd say it is a bad idea because you're not going to succeed at it.

Re: Replace peer review with “peer replication” (2021)

#245

Post by PaulHoule »

RugnirViking wrote: post #179
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

lets be brutally honest with ourselves. 99% of all papers mean nothing. They add nothing to the collective knowledge of humanity. In my field of robotics there are SOOO many papers that are basically taking three or four established algorithms/machine learning models, and applying them to off-the-shelf hardware. The kind of thing any person educated in the field could almost guess the results exactly. Hundreds of suc…

99% is bombastic. What I would say is that the median scientific paper is wrong and back that up with a very long list of things that could make a paper "wrong" or "not even wrong". In the case of physics, everything about string theory may be one day considered "wrong". In the case of medicine all the studies where N < 1/10 the number it would take to draw a reliable conclusion are wrong.

Re: Replace peer review with “peer replication” (2021)

#246

Post by kibibu »

jakear wrote: post #172
vibrio wrote: post #156

Earlier quoted context omitted.

“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.

It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".

There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv.

One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of their capabilities. This wouldn't pass peer review.

Re: Replace peer review with “peer replication” (2021)

#247

Post by brightball »

ImPostingOnHN wrote: post #125
brightball wrote: post #61

Earlier quoted context omitted.

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. The alternative is a bunch of stuff being published which people belief as "science" that doesn't hold up under scrutiny, which undermines the reliability of science itself. The current approach simply gives people reason to be skeptical.

I'm not convinced this proposed alternative is better than the status quo. It's simply not feasible, no matter how many benefits one might imagine. the concern about skepticism is not irrelevant, but many of these skeptics also are skeptical of the earth being round, or older than a few thousand years, or not created by an omnipotent skylord, and I'm not sure it's actually a significant concern given the current numb…

> skeptical of the earth being round

I’m skeptical that these people truly exist outside of the internet wanting it to be true.

Re: Replace peer review with “peer replication” (2021)

#248

Post by gus_massa »

coldtea wrote: post #236
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. Then perhaps those papers shouldn't be published? Or held in any higher esteem than a blog post by the same authors?

An arXiv preprint is like a blog post.

A paper in a peer review journal is like posting a request for reproduction in a heavily moderated mailing list.

A paper in a predatory journal is like the "You are the best ___" price that you get if you pay to go to the "congress" invitation in spam.

Neither of them guaranty that the result is true. The publication in some peer review journals give a minimal guaranty that the paper is not horribly bad, but I've seen too much crap there too.

I know a few journals and author in my area that are serious and I can guess the result will hold, but I find very difficult to evaluate journals and authors in other areas.

Re: Replace peer review with “peer replication” (2021)

#249

Post by dkqmduems »

mapt wrote: post #240
sebzim4500 wrote: post #103

Earlier quoted context omitted.

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

We could easily 10x the funding and 5x the manpower we throw at STEM research if we actually cared what they produced. NSF grants distribute 8.5 billion dollars a year, which is less than Major League Baseball (and its Congressionally granted monopoly) makes. The US Congress has directed 75 billion dollars in aid to Ukraine to date.

That would be reasonable, which isn't the point.

Re: Replace peer review with “peer replication” (2021)

#250

Post by ImPostingOnHN »

brightball wrote: post #247
ImPostingOnHN wrote: post #125

Earlier quoted context omitted.

I'm not convinced this proposed alternative is better than the status quo. It's simply not feasible, no matter how many benefits one might imagine. the concern about skepticism is not irrelevant, but many of these skeptics also are skeptical of the earth being round, or older than a few thousand years, or not created by an omnipotent skylord, and I'm not sure it's actually a significant concern given the current numb…

> skeptical of the earth being round I’m skeptical that these people truly exist outside of the internet wanting it to be true.

> I’m skeptical that these people truly exist outside of the internet wanting it to be true

I'm not, just like I'm not skeptical that climate change denialists truly exist outside of the internet

there's simply no valid argument to warrant skepticism of either, given the ease of locating evidence that both do

Post reply on HN