Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

291–300 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#291

Earlier quoted context omitted.

> skeptical of the earth being round I’m skeptical that these people truly exist outside of the internet wanting it to be true.

> I’m skeptical that these people truly exist outside of the internet wanting it to be true I'm not, just like I'm not skeptical that climate change denialists truly exist outside of the internet there's simply no valid argument to warrant skepticism of either, given the ease of locating evidence that both do

Sure there is. Ease of proof.

I can walk outside and prove gravity.

I can find a million pictures of the earth taken from space, look at a globe and view trade routes that circumnavigate it. I can also look to the sky and see the sun and moon are clearly circular which makes a pretty good case for a pattern.

Climate change or the age of the earth are based on a whole lot more interconnected bits of science that even if you studied your entire life you could not truly say that you understand. You’re putting your trust in layers of science that add up to a certain conclusion (which is good). When people are given good reasons to believe that science and peer reviews aren’t always legitimate it undermines that process of trust building on trust.

Modern aviation is layer upon layer of science building on each other, but I can easily watch a plane takeoff to validate all of those processes.

That’s it in a nutshell. If you can easily replicate it, it’s easy to trust. If you can’t, it’s not…especially when it’s used to drive politics.

Re: Replace peer review with “peer replication” (2021)

#292
post #258

Earlier quoted context omitted.

This is the final stage of capitalism, where market success is conflated with scientific rigor.

Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.

You frequently see this kind of reasoning in medical quackery:

> Butt-candling[1] must work, just look all these happy customers!

But history is replete with ineffective or downright harmful treatments being popular long after the evidence showed them to be ineffective or harmful. Homeopathy is a prime example of this, seeing as those concoctions contain either no active ingredients or (in cases of low dilutions still labeled "homeopathic") contain ingredients picked based on notions of sympathetic magic ("like cures like").

[1] A hopefully fictional example.

Re: Replace peer review with “peer replication” (2021)

#294

Earlier quoted context omitted.

Fraud isn't exceedingly rare :( It only seems that way because academia doesn't pay anyone to find it, reacts to volunteer reports by ignoring it, and the media generally isn't interested. Fraud is so frequent and easy to find that there are volunteers who in their spare time manage to routinely uncover not just individual instances of fraud but entire companies whose sole purpose is to generate and sell fake papers…

The wonderful thing about the western world is that most countries value freedom of the press. The dark side of this is that you can spin up your own “scientific journal” and charge people to publish in it, game the rankings like any common SEO scam, and nobody will stop you because (especially here in the US) you’re exercising your first amendment rights. Then people can fill it with nonsense and even script-generat…

With respect, I think you should research the topic more deeply before assuming that this is some sort of fringe problem that doesn't exist in the "actual" scientific community. The second link I provided is by Nature News and states specifically that the problem affects "prestigious journals" (their words).

Auto-generated papers have been published in journals from the IEEE, Elsevier, Springer Nature and other well known publishing houses. These papers have supposedly passed peer review in western journals that have been around for decades, and have been signed off by professional academics. Invariably no satisfactory explanation for how this happens is provided, with "we got hacked" being a remarkably common claim. Quite how you publish an entire magazine full of fraudulent articles due to one person getting hacked is unclear; actual newspapers and magazines don't ever have this problem.

Here's an example. The Springer Nature journal "Personal and Ubiquitous Computing" was established in 1997 and has its own Wikipedia page:

https://en.wikipedia.org/wiki/Personal_and_Ubiquitous_Comput...

The Editor-In-Chief is a British academic, who also has his own Wikipedia page. So these aren't fly-by-night no-brand nobodies. Yet this journal somehow managed to publish dozens of obviously auto-generated papers, like this one:

https://static-content.springer.com/esm/art%3A10.1007%2Fs007...

"The conversion of traditional arts and crafts to modern art design under the background of 5G mobile communication network"

or

https://static-content.springer.com/esm/art%3A10.1007%2Fs007...

"The application of twin network target tracking and support tensor machine in the evaluation of orienteering teaching"

The papers are just template paragraphs from totally unrelated topics spliced together. Nobody noticed this had happened until months after publication, strongly implying that this journal has no readers at all (this is a common theme in all these stories, they never seem to notice themselves). The editor agreed the papers were nonsense (his words), but blamed peer reviewers. Yet this journal claims to have a large editorial board with over 40 people on it, mostly from universities in the Europe, USA and China.

What's amazing is that this exact same "attack" had happened before. The previous year Springer Nature had to retract over 400 papers which were auto-generated in the exact same way. They learned nothing and appear to treat the problem as a similar level of severity to filtering email spam.

And in the last six months alone we've seen major fraud scandals impacting Stanford (the President no less), Harvard and Yale. These are supposedly elite universities and researchers. Francesca Gino was earning over $1M a year. Yet their fraud is being uncovered by motivated volunteers, not any kind of systematic well funded science police.

So all the signs here point towards fraud being incredibly easy to get away with. Whole journals have literally no readers at all, academia relies on Scooby-Doo levels of policing, and supposedly prestigious brands are constantly having fraud uncovered by random tweeters, undergrads doing journalism as a hobby etc.

Re: Replace peer review with “peer replication” (2021)

#295

Earlier quoted context omitted.

Fraud isn't exceedingly rare :( It only seems that way because academia doesn't pay anyone to find it, reacts to volunteer reports by ignoring it, and the media generally isn't interested. Fraud is so frequent and easy to find that there are volunteers who in their spare time manage to routinely uncover not just individual instances of fraud but entire companies whose sole purpose is to generate and sell fake papers…

I hope you appreciate the futility of trying to prove that fraud is ruining science publication by linking to a bunch of publications that capably detect and point out all the fraud.

What makes you think they detect and point out all the fraud? These sites are hobby sites run by people who just run some very basic text filtering software. If fraud was being reliably detected paper mills wouldn't have businesses, yet these companies seem to be quite common and exist in multiple countries.

And recall that I said all this work pre-dates ChatGPT. Using LLMs to generate scientific papers works great, and you won't be able to find them using regexs.

The journals themselves admit there are serious fraud problems and that they don't know what to do about it. So it's very concerning. The world needs a trustworthy scientific literature.

Re: Replace peer review with “peer replication” (2021)

#296
post #148

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

FYI there is a at least one science journal that only publishes reproduced research: Organic Syntheses "A unique feature of the review process is that all of the data and experiments reported in an article must be successfully repeated in the laboratory of a member of the editorial board as a check for reproducibility prior to publication" https://en.wikipedia.org/wiki/Organic_Syntheses

Started in 1924 and still going strong 100 years later. The gold standard for organic chemistry procedures.

"If you can't reproduce a procedure in Org Syn, it's YOUR fault" - my PhD supervisor

Re: Replace peer review with “peer replication” (2021)

#298

Earlier quoted context omitted.

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort…

This waste of effort by way of duplicating unpublished negative results is a big factor in why replicated results deserve to be rated more highly than results that have not been replicated regardless of the prestige of the researchers or the institutions involved… if no one can prove your work work was correct… how much can anyone trust your work… I have gone down the rabbit hole of engineering research before and 90…

Alternatively, maybe researchers should be encouraged to publish all kinds of results and not just "novel" work. Successes, failures, insights, etc. This idea of how we evaluate researchers is often silly. We're asking people to push the bounds of knowledge and we don't know what will come from it or how long it will take. But we are hyper focused on this short term evaluation.
Post reply on HN