Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Is there a heuristic we might use to identify and flag questionable papers?”

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

11–20 of 109 posts

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#11

Post by avsteele »

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%.

After this I use the following heuristics:

- => Lower belief

+ => higher belief

effects multiply, of course, but each is few %.

++ => sufficient details present that I think I can attempt to replicate without contacting authors (this is rare)

+ => code used is available in public repository (subset of above)

+ => published in med-to-high ranking journal

--- => published in obscure journal

-- => socially desirable outcome

+++ => socially undesirable outcome

-- => non-US, non-EU university or nat. lab

+ => national lab

I mostly read papers in my field (exp. physics), of course. But I began reading a bit more in medical and psyche a few years ago. My views are influenced by books like "Medical Nihilism", and "Science Fictions". Reading COVID papers was also caused me to revise my prior-probabilities downward.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#12

Post by polivier »

A good candidate could be Benford's law [1], which makes predictions about the distribution of digits. If the digits of the numbers found in the results section of the paper deviate too much from this law, it could be a red flag.

[1] https://en.wikipedia.org/wiki/Benford%27s_law

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#13

Post by pyinstallwoes »

It seems that the medium of writing will inevitably reach a point of too much information to process, malicious or not; while simultaneously increasing the risk of silos.

What does a post-writing civilization look like? Is it reasonable to imagine scientific progress without the burden of having to battle the computational capability of sufficiently motivated actors who can easily manipulate authored information spaces?

Writing is essentially a massive attack vector to the collective human consciousness prone to Sybil attacks at an alarming saturation point in a post gpt3 society.

It feels like we need something post-writing.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#14

Post by pyinstallwoes »

avsteele wrote: post #11

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effect…

In a world where belief can be increasingly influenced in undetected ways due to sophisticated information targeting allowing any individual or demographic to be explicitly targeted and optimized against in a relentless attempt to modify behavior, how does one form belief that is defensible?

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#15

Post by ilamont »

A senior editor at a top-tier medical journal told me >5 years ago that they have hired staff dedicated to scrutinizing papers from two countries that have poor reputations when it comes to submissions. It's not just prestige and the career boost; in some countries there are monetary incentives associated with publishing in the top journals.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#16

Post by probably_wrong »

DoreenMichele wrote: post #9
amelius wrote: post #4

“When a measure becomes a target, it ceases to be a good measure” Hence, I'm afraid peer-review is still the only option left.

Peer review has issues too at the moment. From what I've read, it worked when the world was a smaller place, basically. Now, peer review is another broken cog in a machine that grew organically out of past practices and which no one really intended and no one knows how to fix.

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far.

In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this without knowing who you are nor what your institution is. And once your paper is accepted it is freely available on the internet forever.

No human activity is free of issues, but to me double-blind peer review is a fairly effective system. And every year the system is tweaked to try and adapt it to newer developments such as the rise of ArXiv. Most criticism I read uses the term "peer review" to refer to whatever Nature, Science, and/or Elsevier are doing at the moment, but just because they suck it doesn't mean that peer-review as a whole does.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#17

Post by closedloop129 »

pyinstallwoes wrote: post #13

It seems that the medium of writing will inevitably reach a point of too much information to process, malicious or not; while simultaneously increasing the risk of silos. What does a post-writing civilization look like? Is it reasonable to imagine scientific progress without the burden of having to battle the computational capability of sufficiently motivated actors who can easily manipulate authored information spac…

You still have the social graph of the authors. Professors are the gatekeepers of the next generation. Like venture capital, you will have to hustle your way to the top for an introduction if you are an outsider.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#18

Post by exmadscientist »

avsteele wrote: post #11

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effect…

My favorite signal for fields I am unfamiliar with has always been

    ---  => journal declares its ISSN prominently
    ---- => journal prints its ISSN on every page
It has not failed me yet.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#19

Post by DoreenMichele »

probably_wrong wrote: post #16
DoreenMichele wrote: post #9

Earlier quoted context omitted.

Peer review has issues too at the moment. From what I've read, it worked when the world was a smaller place, basically. Now, peer review is another broken cog in a machine that grew organically out of past practices and which no one really intended and no one knows how to fix.

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this…

No, no human activity is free of issues.

But historically when the world was smaller, people writing papers were more likely to "speak the same language," have experiences in common that fostered clear communication, etc.

It's a big, wide world. The sheer variety of experiences these days introduces complications never before seen.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#20

Post by gchamonlive »

abstractalgo wrote: post #6

For manual flagging, you can use Unfold Research to leave a review on the paper itself for anyone to see it, and you can add review tags and a description about what you think is wrong with it: https://twitter.com/UnfoldResearch/status/156099536649284812... Tools like scite ( https://scite.ai/ ) do automatic processing (NLP) of other papers and determine whether a reference relationships supports or rejects the paper…

Could manual flagging be abused by a country or agent against another country or agent in order to maliciously discredit papers from that target?
Post reply on HN