“Is there a heuristic we might use to identify and flag questionable papers?”

Instead of a flagging system ranking the credibility of papers once for everyone, can we have a system where each user decides which other users they trust to evaluate papers, and ignore any signal from untrusted users?

For example, I could L1 trust people to evaluate papers and the trustworthiness of other users, and L2 trust others only to evaluate the trustworthiness of papers. If I notice a spurious signal getting in, I can find out who it came from and prune my list of trusted users.

I imagine I'm not the first to think of this. Anyone know of reading material on the topic?

The article measures the quality of papers produced by a country by the percentage of the world's top 1% cited papers they produced. This seems like it could be gamed.

Mentioned in TFA, the GRIM test for checking a paper’s reported mean of integers:

“”” The GRIM test is straightforward to perform. For each reported mean in a paper, the sample size (N) is found, and all fractions with denominator N are calculated. The mean is then checked against this list (being aware of the fact that values may be rounded inconsistently: depending on the context, a mean of 1.125 may be reported as 1.12 or 1.13). If the mean is not in this list, it is highlighted as mathematically impossible.[2][3] “””

Source: Wikipedia

It looks more like the papers in question are taking a large public dataset and performing some statistical analysis on it, often along the lines of showing a correlation between metrics which represents pollution (like CO2 emissions) and metrics which represent economic growth. The source data is real, and the analysis is probably real too; neither is likely to trip Benford's law. The problem is that there's nothing novel being uncovered by these articles -- it will surprise no one, for example, to hear that China's CO2 emissions have risen in conjunction with their energy usage and economic growth (https://doi.org/10.1007/s11356-020-12217-6). Substituting China for Pakistan, or energy usage for foreign remittances, doesn't make for novel research; it's simply plugging in another set of numbers and turning the crank.

Imagine two papers of equal scientific merit: equally strong or weak evidence, equally rigorous or sloppy reasoning, etc. One of them finds that certain races are genetically inferior. One of them finds that democratic countries have stronger economies.

Which is going to have an easier time getting to publication?

If publication is in the Nazi Journal of Nazi Naziness, the first one will be easier to get published. If for whatever reason you're reading journals there, the second of those papers is the one you should expect to be unusually strong.

If publication is in Nature, they probably don't have a lot of Nazis on their editorial team. If that paper got published there, then probably something about it is unusually good somehow despite the unpleasant conclusion.

Caveats:

1. With a conclusion as horrifying as the one in your original question, though, I think it's fair to reason as follows: Even if the paper gave tremendously strong evidence for genetic differences between races, there's no way any sane editor would publish a paper that says the "... and should be culled" bit. Therefore, this paper was almost certainly railroaded through to publication by a crazy editor. Therefore, the fact that it made it is not evidence that anything about it is any good.

2. It's OK for your expectations of a paper's merits to be affected by how plausible you find its conclusions before you read it. Many "socially undesirable" ideas are that way at least partly because most people find them highly implausible, and if most people do then there's a good chance you do too, and if you do then it's fine for that to lower your expectations for the paper.