Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Is there a heuristic we might use to identify and flag questionable papers?”

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

81–90 of 109 posts

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#81

Post by jstanley »

aabaker99 wrote: post #66

Mentioned in TFA, the GRIM test for checking a paper’s reported mean of integers: “”” The GRIM test is straightforward to perform. For each reported mean in a paper, the sample size (N) is found, and all fractions with denominator N are calculated. The mean is then checked against this list (being aware of the fact that values may be rounded inconsistently: depending on the context, a mean of 1.125 may be reported as…

Good idea (for integer-valued measurements), but you can do it without having to bother calculating all fractions with denominator N.

Let's say the reported mean is called m and the sample size is N. Multiply m and N together. Ideally mN should be almost an exact integer. Find ceil(mN)/N and floor(mN)/N and see if they're the same as the reported m, within precision limits.

Example: m=6.24, N=13

mN = 81.12 (exact)

81/13 = 6.231

82/13 = 6.308

So there is no integer numerator (for denominator 13) that gives 6.24, to 2 decimal places, so 6.24 is a mistake or a lie.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#82

Post by pinko »

non-e-moose wrote: post #76

Count the number of "authors". Anything above 100 is absolutely garbage More then 10 is probably irrelevant and unproveable.

World-class papers with >1000 authors are routinely published by large scientific collaborations like CMS, ATLAS, and LIGO.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#83

Post by jpeloquin »

bittercynic wrote: post #62
gchamonlive wrote: post #20

Earlier quoted context omitted.

Could manual flagging be abused by a country or agent against another country or agent in order to maliciously discredit papers from that target?

Instead of a flagging system ranking the credibility of papers once for everyone, can we have a system where each user decides which other users they trust to evaluate papers, and ignore any signal from untrusted users? For example, I could L1 trust people to evaluate papers and the trustworthiness of other users, and L2 trust others only to evaluate the trustworthiness of papers. If I notice a spurious signal gettin…

Sounds like journal clubs but organized through a specialized tech platform rather than normal communication channels.

(Journal clubs are groups of people, usually working in the same field, who pick & present a paper to the rest of the group on an rotating basis.)

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#84

Post by UncleMeat »

probably_wrong wrote: post #16
DoreenMichele wrote: post #9

Earlier quoted context omitted.

Peer review has issues too at the moment. From what I've read, it worked when the world was a smaller place, basically. Now, peer review is another broken cog in a machine that grew organically out of past practices and which no one really intended and no one knows how to fix.

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this…

What I wish is that peer review did not merge so many things into one step. Peer review is used to test for

1. The validity of the results

2. The novelty of the results

3. The importance of the results

Papers are routinely rejected in peer review not because their methods or results are suspect but because their results are not seen as impactful enough to justify space in a conference or journal. This sucks and introduces all sorts of nasty biases into the community. I'd love for these to be disconnected so one could demonstrate that their paper had been read by other experts and found to be within reason without having to worry that one person on the PC who just doesn't think that a certain research direction is valuable will sink my career.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#85

Post by GuB-42 »

The most obvious heuristic is to look at the reputation of the journal that publishes it.

That's the reason we still have these overpriced journals. The "publishing" part is easy now, it is just a matter of putting a pdf on a server, journals are selling reputation.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#86

Post by exmadscientist »

drited wrote: post #59
exmadscientist wrote: post #18

Earlier quoted context omitted.

My favorite signal for fields I am unfamiliar with has always been --- => journal declares its ISSN prominently ---- => journal prints its ISSN on every page It has not failed me yet.

Is the correct interpretation of the minus signs here that you distrust journals that print ISSN on every page more than you distrust journals that print ISSN prominently? Also could you please expand on the thought process behind this heuristic?

> Is the correct interpretation of the minus signs here that you distrust journals that print ISSN on every page more than you distrust journals that print ISSN prominently?

Yes. At least, that was how I interpreted the grandparent post's scale and how I tried to match it.

> Also could you please expand on the thought process behind this heuristic?

Have... have you ever encountered these journals in the wild? They're trash. But they come up in search results because search engines are dumb, and I don't know the "good" journals in every field, so I can't filter them out myself. Until I click. Then I can!

Usually what's in them is papers from researchers based in countries that have a publication count score or bonus. They'll take some overcomplicated homework problem and solve it or set up some overcomplicated scenario and show that it has property X or have a few diagrams or photos of their current overcomplicated-senior-project research and show that it will do Y when they finish it. All worthless. There is no insight whatsoever.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#87

Post by still_grokking »

wrp wrote: post #28
aaaaaaaaaaab wrote: post #23

Earlier quoted context omitted.

Which two countries? India and China?

Funny story...A few years ago, my online reference hunting turned up a paper from China with three Chinese authors, and also exactly the same paper (down to the formatting and footnotes) from three Indian authors in India.

Plot twist: That were the same three people, but the paper was actually written by someone else.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#88

Post by still_grokking »

overboard2 wrote: post #64
belter wrote: post #40

Earlier quoted context omitted.

"China tops U.S. in quantity and quality of scientific papers" - https://archive.ph/bemMy

The article measures the quality of papers produced by a country by the percentage of the world's top 1% cited papers they produced. This seems like it could be gamed.

Maybe… ;-)

https://news.ycombinator.com/item?id=29061356

https://news.ycombinator.com/item?id=27295906

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#89

Post by exmadscientist »

pinko wrote: post #82
non-e-moose wrote: post #76

Count the number of "authors". Anything above 100 is absolutely garbage More then 10 is probably irrelevant and unproveable.

World-class papers with >1000 authors are routinely published by large scientific collaborations like CMS, ATLAS, and LIGO.

Very true, but if you consider things like "The ATLAS Collaboration" to be one (very composite) author, I think GP's reasoning has some merit. There is a lot of homogeneity in a collaboration author like that, so it's a reasonable thing to do. (Also, speaking from experience, most people in the collaboration paid no attention to the paper, unless it was a Big One.)

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#90

Post by still_grokking »

im3w1l wrote: post #73
amelius wrote: post #55

Earlier quoted context omitted.

> This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". You're simply reformulating what peer means in peer-review. What other way would you suggest? Layman-review?

I had a couple approaches in mind, but I think I got carried away in the moment when I wrote that comment. A radical overhaul may do more harm then good.

But you were right in the first comment!

At the moment people do things "for living", in those usual bureaucratic institutions that all institutions become sooner or later, there are initiatives at power that work against the initial idea behind that institution.

That's why progress in science occurs at best "one funeral at a time", but usually not before a whole generation bites the dust.

This plus bureaucracy seem to almost killed larger parts of science by now.

We have collusion rings milking the system. We have broad pure esoteric nonsense "science" circles. We have more and more utter garbage and scam. And we have holy churches that will send the inquisition on you if you dare to disagree.

I don't know how to fix that. But I guess the initiatives, and especially the cash flows would need some re-engineering. But how concretely, I have no clue. Throwing away everything you have is of course almost never a good idea. But some things need to change. Change in a radical way, I guess.

Post reply on HN