Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Is there a heuristic we might use to identify and flag questionable papers?”

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

21–30 of 109 posts

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#21

Post by anonymous_sorry »

polivier wrote: post #12

A good candidate could be Benford's law [1], which makes predictions about the distribution of digits. If the digits of the numbers found in the results section of the paper deviate too much from this law, it could be a red flag. [1] https://en.wikipedia.org/wiki/Benford%27s_law

Benford's "law" doesn't apply to everything, so it would depend a lot on the field and phenomenon being studied.

And to be honest, once it was known to be used in this way, it would be simple to make fake data comply with the law.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#22

Post by Schroedingersat »

probably_wrong wrote: post #16
DoreenMichele wrote: post #9

Earlier quoted context omitted.

Peer review has issues too at the moment. From what I've read, it worked when the world was a smaller place, basically. Now, peer review is another broken cog in a machine that grew organically out of past practices and which no one really intended and no one knows how to fix.

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this…

Reproduction by n sources with n-k unaffiliated sources would be much better, but that requires a lot more resources for research that could otherwise go to yachts or the latest ponzi scheme.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#23

Post by aaaaaaaaaaab »

ilamont wrote: post #15

A senior editor at a top-tier medical journal told me >5 years ago that they have hired staff dedicated to scrutinizing papers from two countries that have poor reputations when it comes to submissions. It's not just prestige and the career boost; in some countries there are monetary incentives associated with publishing in the top journals.

Which two countries? India and China?

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#24

Post by alimw »

pyinstallwoes wrote: post #13

It seems that the medium of writing will inevitably reach a point of too much information to process, malicious or not; while simultaneously increasing the risk of silos. What does a post-writing civilization look like? Is it reasonable to imagine scientific progress without the burden of having to battle the computational capability of sufficiently motivated actors who can easily manipulate authored information spac…

Mathematical claims may be formalised in a proof assistant.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#27

Post by wrp »

The correct answer is given by Andrew Gelman right at the top: "Unfortunately, no, I don’t know of any heuristic, beyond using tools such as GRIM to check for consistency of reported numerical results."

The question is essentially whether it is possible to have a structural procedure to evaluate the semantic validity of a statement.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#28

Post by wrp »

aaaaaaaaaaab wrote: post #23
ilamont wrote: post #15

A senior editor at a top-tier medical journal told me >5 years ago that they have hired staff dedicated to scrutinizing papers from two countries that have poor reputations when it comes to submissions. It's not just prestige and the career boost; in some countries there are monetary incentives associated with publishing in the top journals.

Which two countries? India and China?

Funny story...A few years ago, my online reference hunting turned up a paper from China with three Chinese authors, and also exactly the same paper (down to the formatting and footnotes) from three Indian authors in India.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#29

Post by schiptsovgithub »

Basically, everything with is based on any statistical inference.

Statistics applied to not fully observable stable systems in the modern day alchemy.

Simulations have similar flaws - once the model does not match reality perfectly one would get an arbitrary wrong results.

Look at all the COVID related simulations and statistical models - all of them are not even close to reality.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#30

Post by a_bonobo »

Why are these questionable papers? Academics are forced to publish as many papers as possible. The system pushes their own to go for the smallest publishable unit, people will find a way to optimise for that, and who are we to blame them for that? They're underpaid, overworked researchers trying to guess what the opaque and brutal funding system wants from them.

Is there outright fraud, or are they just dull/pointless papers? if first, yeah that's questionable, but a lot of these example papers look more like they're dull, rather than committing fraud.

We should rather tear down the awful system instead of playing whack-a-mole with academia's newest outcropping.

Post reply on HN