Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

231–240 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#231

Post by dongping »

freeopinion wrote: post #209
dongping wrote: post #195

Earlier quoted context omitted.

In the sense of replicating the results, we do have CI servers and even fuzzers running for our "code replication".

I don't want to derail the science discussion too much, but what if you actually had to reproduce the code by hand? Would that process produce anything of value? Would your habit of writing i+=1 instead of i++ matter? Or iteration instead of recursion? Would code replication result in fewer use after free, or off by one than code review? Or would it mostly be a waste of resources including time?

I'm not sure if it is meaningful to divert the topic to an analogy that is never precise. But I think we always rerun the same code (equivalent to the same procedure in the papers) in the CI or your workstation to debug. Replicating the result of the program doesn't mean that one would have to rewrite the code, as in replicating the result of a computer vision paper may only include code review and running the code from the paper.

Re: Replace peer review with “peer replication” (2021)

#232

Post by matthewdgreen »

mike_hearn wrote: post #191
matthewdgreen wrote: post #60

Earlier quoted context omitted.

No subset of humanity is “pure hearted.” Fraud and malice will exist in everything people do. Fortunately these fraudulent incidents seem relatively rare, when one compares the number of reported incidents to the number of publications and scientists. But this doesn’t change anything. The benefit of scientific publication is to make it easier to detect and verify incorrect results , which is exactly what happened in…

Fraud isn't exceedingly rare :( It only seems that way because academia doesn't pay anyone to find it, reacts to volunteer reports by ignoring it, and the media generally isn't interested. Fraud is so frequent and easy to find that there are volunteers who in their spare time manage to routinely uncover not just individual instances of fraud but entire companies whose sole purpose is to generate and sell fake papers…

The wonderful thing about the western world is that most countries value freedom of the press. The dark side of this is that you can spin up your own “scientific journal” and charge people to publish in it, game the rankings like any common SEO scam, and nobody will stop you because (especially here in the US) you’re exercising your first amendment rights. Then people can fill it with nonsense and even script-generated fake papers. People outside the scientific community can also scam more “legitimate” for-profit journals in various ways, resulting in more silly publications that the actual scientific community has to filter out. It’s very annoying.

None of this has any more bearing on fraud by professional scientists than, say, the existence of some garbage-filled Wikimedia server or a badly-edited Wikipedia page means that the Wikipedia editors themselves are fraudsters.

Re: Replace peer review with “peer replication” (2021)

#233

Post by jonnycomputer »

If they are, in fact, implying that another lab should produce a matching data-set to try to replicate results, well, I'm sorry, but that won't work, at least in a whole lot of fields. Data collection can be very expensive, and take a lot of time. It certainly is in my field.

If on, the other hand, they just want the raw data, and let others go to town on it in their own way, that's fine, probably. Results that don't depend on very particular details of the processing pipeline are probably more robust anyway.

Re: Replace peer review with “peer replication” (2021)

#234

Post by iamthemonster »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

My Master's thesis was basically taking a purely theoretical paper and "replicating" it, by which I mean taking the formulae and just writing the software to run them. It sounds trivial to an outsider but even that was I guess 300 hours of work.

In general I think undergraduate projects are a great space to attempt to replicate findings, but it heavily depends on the field. Fundamental physics experiments can be expensive and require equipment that's outside the reach of undergrads. But one thing I love about engineering as an academic field, by comparison, is that anything you research tends to be more achievable for others to replicate because as your end goal you are aiming for something that's practical in the field.

Re: Replace peer review with “peer replication” (2021)

#235

Post by peteradio »

jonnycomputer wrote: post #233

If they are, in fact, implying that another lab should produce a matching data-set to try to replicate results, well, I'm sorry, but that won't work, at least in a whole lot of fields. Data collection can be very expensive, and take a lot of time. It certainly is in my field. If on, the other hand, they just want the raw data, and let others go to town on it in their own way, that's fine, probably. Results that don't…

What field is it too expensive or difficult to reproduce the data?

Re: Replace peer review with “peer replication” (2021)

#236

Post by coldtea »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper.

Then perhaps those papers shouldn't be published? Or held in any higher esteem than a blog post by the same authors?

Re: Replace peer review with “peer replication” (2021)

#237

Post by LBTables »

RugnirViking wrote: post #179
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

lets be brutally honest with ourselves. 99% of all papers mean nothing. They add nothing to the collective knowledge of humanity. In my field of robotics there are SOOO many papers that are basically taking three or four established algorithms/machine learning models, and applying them to off-the-shelf hardware. The kind of thing any person educated in the field could almost guess the results exactly. Hundreds of suc…

> In my field of robotics there are SOOO many papers that are basically taking three or four established algorithms/machine learning models, and applying them to off-the-shelf hardware.

This is a direct result of the aggressive "publish or perish" system. I worked as an aide in an autonomous vehicles lab for a year and a half during my undergrad, and while the actual work we were doing was really cool cutting edge stuff, it was absolutely maddening the amount of time we wasted blatantly pulling bullshit nothing papers exactly like you describe out of our asses to satisfy the constant chewing out we got that "your lab has only published X papers this month".

Re: Replace peer review with “peer replication” (2021)

#238

Post by 542458 »

peteradio wrote: post #235
jonnycomputer wrote: post #233

If they are, in fact, implying that another lab should produce a matching data-set to try to replicate results, well, I'm sorry, but that won't work, at least in a whole lot of fields. Data collection can be very expensive, and take a lot of time. It certainly is in my field. If on, the other hand, they just want the raw data, and let others go to town on it in their own way, that's fine, probably. Results that don't…

What field is it too expensive or difficult to reproduce the data?

As reviewers are paid nothing and get no substantial credit for their work, I’m going to say “Every field”. Why would you do a significant chunk of a paper’s work for no reward? Replication studies are typically a bad deal even when you get to pick a notable study to reproduce and you get a paper to your name out of it - replicating what will probably be an obscure paper for no credit is not something most academics (let alone commercial research labs) would entertain.

Re: Replace peer review with “peer replication” (2021)

#239

Post by ugh123 »

Why not just develop a standard "replication instructions" format that papers would need to adhere to? All methods, source code, ingredients, processes, etc are documented in a standard way. This could help tease out a lot of bullshit just by reading this section.

Re: Replace peer review with “peer replication” (2021)

#240

Post by mapt »

sebzim4500 wrote: post #103
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.

We could easily 10x the funding and 5x the manpower we throw at STEM research if we actually cared what they produced.

NSF grants distribute 8.5 billion dollars a year, which is less than Major League Baseball (and its Congressionally granted monopoly) makes. The US Congress has directed 75 billion dollars in aid to Ukraine to date.

Post reply on HN