Live data from Hacker News

I tried to report scientific misconduct. How did it go?

crystalprisonzone.blogspot.com

141–150 of 363 posts

Re: I tried to report scientific misconduct. How did it go?

#141
> Science is supposed to be self-correcting.

I admire the author's belief, not least because I used to be like that, but I personally think that couldn't be further form the truth for contemporary scientific research, and it's no better in evidence-base physical sciences. I personally know many people who used to be, or still are, in scientific research who wouldn't hesitate to agree with me that scientific research is mostly just a job for most people that's not too different to any other job that earns you a salary.

I always ended up not posting my comment in related topics, but since this is getting so much traction, I might as well try not to appear to be bitter about my own experience and give my anecdote another go. If nothing else, at least this will become (albeit insignificant) a piece of history that stays on the Internet.

I long time ago I received a prestigious postdoctoral fellowship to work with someone very well-known in the field on studying the mechanisms of a then relatively new type of chemical reactions. I spent a couple of months to meticulously prepare everything I needed for the study, and when finally I got everything ready, I began by reproducing the first break-through that was produced in the group that started it all -- and it didn't work.

Since I was new to that particular type of chemistry at that time, I spent the next few months trying to reproduce the reaction while getting others, both within and without the group, to check my work. Nobody seemed to be able to figure out what I did wrong but one particularly thing stood out at that time: nobody I have spoken to actually tried to reproduce the results of the "first" reaction, ever, which was super strange to me. I had also spoken with my advisor then, who basically became well-known because of that first reaction, and he couldn't offer any solutions and the conversation always ended up being something completely unrelated to the irreproducible results. I spent most of that time blaming myself and suffering from some form of imposter syndrome, too, simply because I have the tendency to do that.

Up till that point I had been following the procedure published in a journal article, but I thought I would dig up the first author's PhD thesis to check what I had done wrong. I started by casually scrolling through the experimental section and an C-13 NMR spectrum of the catalyst that I was working with caught my eye immediately because of some very unnatural signal truncation that I thought was only possible with data manipulation, and sent the data to a few of my friends who are experts in NMR and they also confirmed that those "artifacts" are most certainly unnatural. I immediately e-mailed my advisor about it, but he never responded -- and that was the only e-mail from me (which obviously required a response) that he never responded to.

I did find a few manipulated spectra in the same PhD thesis, but none of that really helped because I still couldn't reproduce the results that nobody has ever mentioned anything wrong about. Then one night, when I was drinking with the group, someone working on a different floor I don't usually talk to about my work asked me how things were going; after I told him my problems he immediately said that he'd met someone from industry at a conference complaining to him that the reaction doesn't work. He also said that a few people who came before me also tried to reproduce that reaction but none of them got it to work.

At that point I was just angry because *I thought "science is supposed to be self-correcting"* and there is no way that this stuff was in the literature for 10 years and nobody ever said anything about it. In fact, it's impossible for my advisor to not know that something is wrong with it because he is very well connected to both academia and industry, and so many people in the 10 years before I arrived must have worked on it.

During the time I was unable to get anything to work, I was constantly assigned work that seem somewhat related to what I do but wouldn't help me with my career in any way. In the end I had a hunch on what was really happened and determined that the procedures in the original paper and the PhD thesis that first reported the reaction were all out by a factor of 10. I was already on anti-depressants at that point and was drunk every night but was working 10+ hours a day, which was well-known in the group. When I had finally gotten the reaction to "work" (and had explained to people I trust and had them double check my work) and brought it to my advisor, he said "that's great"; I don't remember too well what else he'd said in between because none of it was neither an apology nor a solution, but he said at the end that maybe I should have deferred my fellowship because of my depression (which, frankly, wasn't affecting my ability to work).

This is not an isolated case, and not the only type of academic misconduct. The thing that upsets us the most is that at the end of the day, it's not about how good and meticulous you are: for most of us it's mostly about how well you are at gaming the system. The way we fund scientific research is mostly broken, the way we disseminate scientific research is mostly broken, the way we assess potentially great scientists and appoint them is also mostly broken. It's only natural that, for most people, the experience is nothing but shit.

Edit: typo.

Re: I tried to report scientific misconduct. How did it go?

#142

Earlier quoted context omitted.

My cousin was a student at a lab where a sketchy grad student doctored results too. She was majorly sketched out by the whole thing and that the PI supported the whole op. It was very painful and set her back a bit but she managed to switch to a different lab and do proper work, defend her thesis, graduate, and get far away from those people. Now we just shake our head in disbelief at them but at that point it was fa…

Every now and then I go through some of wikipedias sources on certain social topics because I don't trust them at all. The amount of BS I've found in papers, even though I don't eve have any research background at all is impressive. My favourite was probably this one paper where the author essentially made a reddit-post asking a community about themselves, then cherry-picked (the post is still up, with timestamps and…

Did you edit the Wikipedia page or challenge the interpretation in the talk page?

Re: I tried to report scientific misconduct. How did it go?

#143
We are dealing with 5+ papers that are fraudulent and another 5+ newer papers that are most likely fraudulent, too. That is, there have been 20, 25+ reviewers looking at those papers. Their job was to carefully read them and double check the numbers. All of them gave those papers a pass. I am at a loss here.

The authors' behaviour is outrageous, but this story is also about a broken reviewing process, partly due to wrong incentives.

Re: I tried to report scientific misconduct. How did it go?

#144

Earlier quoted context omitted.

I think the point of GP's story is that this isn't just one or two bad apples here and there, but it's endemic in that domain - and most likely in others too (I'm leaning to believe that; it's not the first story like this I've read in recent years). Being an endemic problem means you have to switch your assumptions; when reading a random scientific paper, you're no longer thinking, "this is probably right, but I mus…

I think once you've seen a few papers in high-tier journals that turn out to be bullshit once you start to dig a bit deeper, there is not other choice than to adopt this harsh stance on random scientific papers. Especially if you want to do work with that expands on findings on other papers that roughly look good "trust but verify" seems to be the way to go. I've only recently dipped by toes into academic life in a l…

> I think once you've seen a few papers in high-tier journals that turn out to be bullshit once you start to dig a bit deeper, there is not other choice than to adopt this harsh stance on random scientific papers. Especially if you want to do work with that expands on findings on other papers that roughly look good "trust but verify" seems to be the way to go.

Yeah, but I meant that in general case, you no longer "trust but verify", but "assume bullshit and hope there's a nugget of truth in the paper".

This has interesting implications for consuming science for non-academic use, too. I've been accused of being "anti-science" when I said this before, but I no longer trust arguments backed by citations around soft-ish fields like social sciences, dietetics or medicine. Even if the person posting a claim does good work of selecting citations (so they're not all saying something tangentially related, or much more specific, or "in mice!"), if the claim is counterintuitive and papers seem complex enough, I mentally code this as very weak evidence - i.e. most likely bullshit, but there were some papers claiming it, so if that comes up again, many times in different contexts, I may be willing to entertain the claim being true.

And stories like this make me extend this principle to biology and chemistry in general as well. I've burned myself enough times, getting excited about some result, only to later learn it was bunk.

The same pattern of course repeats outside academia, but more overtly - you can hardly trust any commercial communication either. At this point, I'm wondering how are we even managing to keep a society running? It's very hard work to make progress and contribute, if you have to assume everyone is either bullshitting, or repeating bullshit they've heard elsewhere.

Re: I tried to report scientific misconduct. How did it go?

#145
post #82

What if you had a webpage with the paper at the top and a comments section at the bottom where anyone in the world could post a rich-media post of their findings that cast doubt on some findings with regard to the original paper. We currently treat academic research by a metric of "citations", why not have a new metric where the world can be cautioned of potential issues with the result. You could use the bug bounty…

> ...anyone in the world could post... This would be ultimately filled with trolls, nonexperts, and people with an ulterior motive or grudge: so any such effort would be less trustworthy than the original publication. If you want to find mistakes, you can find one in every paper. The character of the mistakes is important - is it fraud, is it incompetence, does it negate the results? Or is it good science being done…

> any such effort would be less trustworthy than the original publication.

The idea that scrutiny would be less reliable than blind trust is absurd. The question in the OP, for example, could have been in the comments sections of these papers.

Re: I tried to report scientific misconduct. How did it go?

#146
post #24

Earlier quoted context omitted.

Reminds me of " a lie can run around the world before the truth has got its boots on ". That precise quote is from Pratchett but there are similar, earlier citations https://quoteinvestigator.com/2014/07/13/truth/

Aren't these all variations on well known aphorisms such as "publish or perish"?

[deleted]

Re: I tried to report scientific misconduct. How did it go?

#147

Earlier quoted context omitted.

I'm sorry but this saddens me to no end, even I did better science during my BSc and MSc; it's not just disheartening, it's frightening. Reading this almost made me feel ill to my stomach. I don't know what else to say at the blatant disregard for scientific ethics and sense of duty. And we complain that the public at large doesn't trust us "educated" folk, well I can't see why...

My own, rather bitter, experiences with academic research in the early 1990s led me to suspect that by trying to "manage" academic research at a large scale was utterly counter productive and was optimising for all the wrong things (publications, career progression, money, politics) and was actually dramatically reducing the amount of actual science being done. I left, co-founded a startup and never regretted it for…

What does conservative mean in this context? Could you explain it a bit? Thanks!

Re: I tried to report scientific misconduct. How did it go?

#148

Earlier quoted context omitted.

> ...anyone in the world could post... This would be ultimately filled with trolls, nonexperts, and people with an ulterior motive or grudge: so any such effort would be less trustworthy than the original publication. If you want to find mistakes, you can find one in every paper. The character of the mistakes is important - is it fraud, is it incompetence, does it negate the results? Or is it good science being done…

> any such effort would be less trustworthy than the original publication. The idea that scrutiny would be less reliable than blind trust is absurd. The question in the OP, for example, could have been in the comments sections of these papers.

I understood that this was about scrutinising papers in academic journals - the academic journal's value is quite literally its trustworthiness. (The journal exists and is employed to do the scrutiny)

A comment from any random person (in general) holds a lower level of trustworthiness.

Re: I tried to report scientific misconduct. How did it go?

#149

Earlier quoted context omitted.

How does bad research get retracted and corrected then? As the author points out, the "data" in these papers is large enough to contaminate meta-analyses for years to come. And if the Bad Scientist continues to produce more of them, then decades to come. The consensus of the entire discipline will be swayed. Self-correcting this will be very difficult, require lots of data, and be unrewarding. It probably won't happe…

The usual way is someone writing a better paper and everyone going "yeah, that's a better reading of the data". Not sure why you say correcting an established consensus would be unrewarding? Sure, for unimportant details getting a correction out isn't much fun, but correcting an important point is basically the career goal of every scientist precisely because it is important and rewarding.

From the author, who specifically stated that this task of correcting bad science is unrewarding.

Re: I tried to report scientific misconduct. How did it go?

#150
post #141

> Science is supposed to be self-correcting. I admire the author's belief, not least because I used to be like that, but I personally think that couldn't be further form the truth for contemporary scientific research, and it's no better in evidence-base physical sciences. I personally know many people who used to be, or still are, in scientific research who wouldn't hesitate to agree with me that scientific research…

Basing your own work on something that doesn't work is incredibly frustrating and can lead to enormous amounts of wasted effort. It doesn't even have to be fraud, there are so many factors you often can't fully control, and reactions can depend on very subtle details or minor impurities.

My impression is that usually the informal communication about stuff "that everyone knows doesn't actually work" is far more efficient than in your case. But this is something the PI has to do, as a new PhD student won't be connected enough for this, and your PI seriously failed you there.

It would be nice if someone published that this method doesn't work, but that doesn't seem to be how this works. The amount of effort to actually demonstrate that it really doesn't work is so much higher than the reward.

In a healthy environment people should have been much more sceptical much earlier. At the latest when you saw potential manipulations in the NMR. I'm curious what kind of artifacts you saw there, did they just remove or add signals?

Post reply on HN