Live data from Hacker News

I tried to report scientific misconduct. How did it go?

crystalprisonzone.blogspot.com

21–30 of 363 posts

Re: I tried to report scientific misconduct. How did it go?

#21
post #9
post #7

"Social psychology" has had a serious problem with scientific misconduct and statistical incompetence. Is this related in some way to that subject area?

What about the rest of academic psychology for that matter? Is this endemic to that field?

The field is over-crowded, for one. Two, its participant pool is almost exclusively undergraduate students on university campuses in WEIRD (western, educated, industrialized, rich, and democratic) nations. Three, experimental designs may rely on questionably reliable, possibly retrospective self-report to infer patterns of affect and cognition. Four, experiments are often conducted in highly contrived laboratory environments. (In-situ data collection as people go about their lives is becoming more common with the rise of smartphones and wearables, however.)

As in most domains of science, researchers are also pressured to publish regardless of the quality of the work, and replicated findings are usually not considered interesting or valuable enough to publish.

This is anecdata, so take it with a grain of salt. (I minored in psychology and worked as a research assistant with a social/cognitive psych lab as an undergrad.) Also, there is a ton of extremely good science in psychology. It's a shame the recent profusion of low-quality work obscures it.

Re: I tried to report scientific misconduct. How did it go?

#23

Earlier quoted context omitted.

> Research output is reliable where it is actively relied on by engineers -- and not elsewhere. I'd word it this way: if the person producing the output is not responsible for it really working, it almost certainly won't. Even innocently this will be the case with anything complex, people get things backwards, miss a scale factor, etc. Finding that last bug can take more work than the rest of the project combined, mu…

> if the person producing the output is not responsible for it really working, it almost certainly won't. Even innocently this will be the case with anything complex Indeed, there's a great example in the article itself, in a totally unrelated area: > I felt these journals generally did their best, and the slowness of the process likely comes from the bureaucracy of the process and the inexperience editors have with…

> these reasonable journals weren't able to use their retraction process even though they wanted to, because the process never gets used and therefore isn't in a usable state.

Actually, this made me think about a journal purposefully running intentionally spurious papers, with a challenge to the readership to identify which paper was fake. If the system worked, that would cause every paper published in the journal to be investigated adversarially.

The obvious problem with the idea seems to be that so much of the process is voluntary; people might be unwilling to submit papers to that journal.

Re: I tried to report scientific misconduct. How did it go?

#24

The article mentions Brandolini’s Law: “The amount of energy needed to refute bullshit is an order of magnitude larger than to produce it.” It's the first time I've heard it, but it's a very appropriate observation in today's world where misinformation travels faster and wider than correct information. If you're just making stuff up, it's much faster than looking up sources.

Reminds me of " a lie can run around the world before the truth has got its boots on ". That precise quote is from Pratchett but there are similar, earlier citations https://quoteinvestigator.com/2014/07/13/truth/

Aren't these all variations on well known aphorisms such as "publish or perish"?

Re: I tried to report scientific misconduct. How did it go?

#25

This is pretty interesting stuff, but one note: > The correction explains away the failures of randomization as an error in translation; the authors now claim that they let participants self-select their condition. This is difficult for me to believe. The original article’s stressed multiple times its use of random assignment and described the design as a "true experiment.” > They also had perfectly equal samples per…

> This actually cannot happen with random assignment either. The only way you're going to get equal numbers in each bin is if your process is intentionally constrained to do that.

There are several CS shuffle/Fisher-Yates algorithms that can do this. Instead of calling the usual rand() on a mathematical interval multiple times, they do selection over the remaining elements (ie. constrained.)

https://dev.to/babak/an-algorithm-for-picking-random-numbers...

But I would expect CS people to have awareness about that, not social scientists, unless somebody wrote a paper with examples for that field.

I've seen Fisher-Yates used in an SRE interview before, which is pedantic - it's just whiteboard hazing, at a very high cost to your recruiting and interviewing staff.

Re: I tried to report scientific misconduct. How did it go?

#26
post #22

I really like this work. Her methodology is remarkably simple. Which begs the question, is peer review so broken that simple things like this can't be caught? Should we have a government ministry of academic papers?

that's kind of what we have already, though responsibility is divided between DARPA, NSF, NIH, DOE, NCI.

Re: I tried to report scientific misconduct. How did it go?

#27

This is pretty interesting stuff, but one note: > The correction explains away the failures of randomization as an error in translation; the authors now claim that they let participants self-select their condition. This is difficult for me to believe. The original article’s stressed multiple times its use of random assignment and described the design as a "true experiment.” > They also had perfectly equal samples per…

Here, "random assignment" means randomly assigning half the participants to each of two bins, I think.

Re: I tried to report scientific misconduct. How did it go?

#29
post #24

Earlier quoted context omitted.

Reminds me of " a lie can run around the world before the truth has got its boots on ". That precise quote is from Pratchett but there are similar, earlier citations https://quoteinvestigator.com/2014/07/13/truth/

Aren't these all variations on well known aphorisms such as "publish or perish"?

I think the ones above are much more general than 'publish or perish'.

Re: I tried to report scientific misconduct. How did it go?

#30

I think there is at least one or two platforms to rate and review articles after they are published. But I can't find them. Does anyone remember their names ?

Are you thinking of PubPeer? https://pubpeer.com/

Which I recommend the browser extension for. It provides a banner at the top of the page anytime the site displays a paper with PubPeer comments on it. https://addons.mozilla.org/en-US/firefox/addon/pubpeer/

Plenty of comments when you visit this thread's site. :)

Post reply on HN