Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Is there a heuristic we might use to identify and flag questionable papers?”

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

51–60 of 109 posts

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#51

Post by Bakary »

avsteele wrote: post #11

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effect…

Could you expand on what you define as socially desirable or undesirable? I understand the point you are making but I am not sure exactly through what lens you are quantifying these characteristics.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#52

Post by Bakary »

pyinstallwoes wrote: post #14
avsteele wrote: post #11

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effect…

In a world where belief can be increasingly influenced in undetected ways due to sophisticated information targeting allowing any individual or demographic to be explicitly targeted and optimized against in a relentless attempt to modify behavior, how does one form belief that is defensible?

There are a few rules of thumb you can apply:

- Only a tiny minority of your beliefs have any impact on the outcome of your life. Perhaps your concept of [insert social or political policy] or [scientific topic] or [cultural idea] might have been gamed by someone at some point, but ultimately it will rarely matter whether you even hear of these concepts to begin with or not.

- Of the beliefs that remain, these often become pretty defensible through practice and results.

Put another way, there are plenty of barely-literate people who still have lead fascinating and worthwhile lives even in the modern era.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#53

Post by peteradio »

seydor wrote: post #41

indeed there is no such heuristic. The question is deeper, it's about how to trust that scientists from a culture that is perceived to be alien are living up to familiar scientific standards. It's easier to identify fraud when it comes in a systematic way from a specific region of the world. However there is a lot of fraud and scientific-freeriding coming from western scientists too, it's just sporadic and not visibl…

I wonder are there tools to visualize a particular area of interest. Could be used to identify duplicate results (not a bad thing), useless results, or contradicting results (possible fraud, error or even ground breaking). When I read too deeply in almost any area I lose the forest for the trees, authors are great at rigorously describing in english what they are doing but it overflows my stack. I wish there was a way to map it semi-permanently so I could say aha yes this result looks like this on my map of the domain. Does such a general tool exist?

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#55

Post by amelius »

im3w1l wrote: post #31
probably_wrong wrote: post #16

Earlier quoted context omitted.

As someone who does peer review for 2-3 conferences a year, I will say that peer review is still better than any alternative proposed so far. In my circles (Computational Linguistics), getting your paper approved means that you convinced at least three PhD students with a published paper (or higher, all the way up to Professor) that your paper is good PLUS the area chair(s) considered it worth publishing. All of this…

> getting your paper approved means that you convinced at least three PhD students with a published paper This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)". If you have a good initial condition maybe it can work for a while, but it seems that time may be coming to an end. > Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will…

> This is a recursive definition of scientist "A scientist(t=n) is someone approved by scientists(t=n-1)".

You're simply reformulating what peer means in peer-review.

What other way would you suggest? Layman-review?

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#56

Post by kQq9oHeAz6wLLS »

psyc wrote: post #50

Just ask yourself if you dislike the conclusions or implications. If so, the paper isn't credible. I was under the impression we knew this method for a pretty long damn time by now.

Unfortunately, the reverse will also be true: if you like the conclusions or implications, it must be credible.

This is confirmation bias, and has been used for terrible reasons, as well as to great comedic effect (see Sea Matheson fat studies)

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#57

Post by probably_wrong »

RcouF1uZ4gsC wrote: post #25

> The papers come from dozens of different researchers primarily not in North America or Europe, That seems to be the actual heuristic, but it would not fly politically.

This would be a terrible heuristic, as it would reward western bad science (it exists, it just looks differently) and punish great scientists who just happen to be born in the "wrong" place.

Paradoxically enough, it would be the opposite of science.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#58

Post by avsteele »

Bakary wrote: post #51
avsteele wrote: post #11

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effect…

Could you expand on what you define as socially desirable or undesirable? I understand the point you are making but I am not sure exactly through what lens you are quantifying these characteristics.

https://www.econlib.org/archives/2014/04/social_desirabi_1.h...

The proximal desirability bias is that of publication gatekeepers. Biases of society at large also can play a significant role for certain topics.

Re: “Is there a heuristic we might use to identify and flag questionable papers?”

#59

Post by drited »

exmadscientist wrote: post #18
avsteele wrote: post #11

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effect…

My favorite signal for fields I am unfamiliar with has always been --- => journal declares its ISSN prominently ---- => journal prints its ISSN on every page It has not failed me yet.

Is the correct interpretation of the minus signs here that you distrust journals that print ISSN on every page more than you distrust journals that print ISSN prominently?

Also could you please expand on the thought process behind this heuristic?

Post reply on HN