Live data from Hacker News

“Is there a heuristic we might use to identify and flag questionable papers?”

statmodeling.stat.columbia.edu

31–40 of 109 posts

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#31

Earlier quoted context omitted.

Peer review has issues too at the moment. From what I've read, it worked when the world was a smaller place, basically. Now, peer review is another broken cog in a machine that grew organically out of past practices and which no one really intended and no one knows how to fix.

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this…

> getting your paper approved means that you convinced at least three PhD students with a published paper

This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end.

> Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: First, there will be those who are devoted to the goals of the organization. Examples are dedicated classroom teachers in an educational bureaucracy, many of the engineers and launch technicians and scientists at NASA, even some agricultural scientists and advisors in the former Soviet Union collective farming administration. Secondly, there will be those dedicated to the organization itself. Examples are many of the administrators in the education system, many professors of education, many teachers union officials, much of the NASA headquarters staff, etc. The Iron Law states that in every case the second group will gain and keep control of the organization. It will write the rules, and control promotions within the organization.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#32

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effect…

You would assign a +++ score to a paper that says, for example, that certain races are genetically inferior and should be culled??

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#33

Why are these questionable papers? Academics are forced to publish as many papers as possible. The system pushes their own to go for the smallest publishable unit, people will find a way to optimise for that, and who are we to blame them for that? They're underpaid, overworked researchers trying to guess what the opaque and brutal funding system wants from them. Is there outright fraud, or are they just dull/pointles…

There is outright fraud but I think most are just dull and pointless.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#34

Why are these questionable papers? Academics are forced to publish as many papers as possible. The system pushes their own to go for the smallest publishable unit, people will find a way to optimise for that, and who are we to blame them for that? They're underpaid, overworked researchers trying to guess what the opaque and brutal funding system wants from them. Is there outright fraud, or are they just dull/pointles…

> We should rather tear down the awful system instead of playing whack-a-mole with academia's newest outcropping.

There isn’t one “system” that can be torn down and replaced. A lot of these papers are coming from groups and institutions in foreign countries that are basically trying to role-play as researchers so they can blend in with actual researchers and try to reap the reputation and career rewards.

It’s not really centralized. This is akin to suggesting that we need to tear down the entire “e-mail system” because spammers are abusing it to send spam appearing as legitimate email.

Why is it a problem? Because these contribute nothing but noise and confusion to the field and make it more difficult for actual researchers to sort through real research. There is no reason to forgive these faux-researchers for trying to extract money and prestige from a system without making actual contributions.

The idea to “tear down the system” doesn’t even make sense unless you propose something different. Are you suggesting we stop rewarding anyone in any way for research? That would remove the incentives, but now we have a problem that research isn’t rewarded at all and real researchers won’t do it because they need to eat.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#35

Why are these questionable papers? Academics are forced to publish as many papers as possible. The system pushes their own to go for the smallest publishable unit, people will find a way to optimise for that, and who are we to blame them for that? They're underpaid, overworked researchers trying to guess what the opaque and brutal funding system wants from them. Is there outright fraud, or are they just dull/pointles…

Is there also danger that any flagging system could be abused, with the end of censorship of problematic viewpoints, rather than merely bad science?

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#36
post #31

Earlier quoted context omitted.

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this…

> getting your paper approved means that you convinced at least three PhD students with a published paper This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. > Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will…

> This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end.

There is no formal definition for a "scientist" and likely will never be. Definitely a scientist is not "one who has a paper accepted".

Here the commenter is just claiming that in order to publish a paper in a specific journal you do need be accepted by a handful of people who have previously been accepted in the same journal. Barring the obvious chicken and egg problem which is easily solved, it does not seem to be like something which "is coming to an end".

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#37

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effect…

Take logs, then the effects add.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#39

For manual flagging, you can use Unfold Research to leave a review on the paper itself for anyone to see it, and you can add review tags and a description about what you think is wrong with it: https://twitter.com/UnfoldResearch/status/156099536649284812... Tools like scite ( https://scite.ai/ ) do automatic processing (NLP) of other papers and determine whether a reference relationships supports or rejects the paper…

Could manual flagging be abused by a country or agent against another country or agent in order to maliciously discredit papers from that target?

Most definitely.

An advantage of having open peer review system is also that this kind of attempts could be detected by tools doing automatic processing of scientific knowledge graphs generated by peer review activity.

This, coupled with verification of users, should be able to minimize the risk quite significantly.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#40
post #15

A senior editor at a top-tier medical journal told me >5 years ago that they have hired staff dedicated to scrutinizing papers from two countries that have poor reputations when it comes to submissions. It's not just prestige and the career boost; in some countries there are monetary incentives associated with publishing in the top journals.

Which two countries? India and China?

"China tops U.S. in quantity and quality of scientific papers" - https://archive.ph/bemMy
Post reply on HN