Live data from Hacker News

Cracking down on research fraud

undark.org

31–40 of 227 posts

Re: Cracking down on research fraud

#31

Earlier quoted context omitted.

Thanks for pointing out the problem with credentials. Name a better indicator then for being competent.

Industry kinda already has. Products launched. Teams managed. A candidate's publication record is a conversation starter during interviews, but I feel like years of experience at a company working on launching a successful project is a much more valuable currency.

> Industry kinda already has. Products launched.

This suggestion is so detached from both the problem and the very nature of academia that it's straight-out laughable.

I mean, what do product launches have to do building knowledge on the state of the art, identifying a novel idea, doing the iterative work to refine the idea, and finally document it to the public? Product launches at best require you to manage people and expectations. Do you honestly believe that a guy who launched a product is more qualified to drive science forward than a PhD with an outstanding academic track record just because your PM had a knack to cut corners, descope requirements, and pass the buck to underlings? Because that's the bulk of the job of all PMs I ever met, including from FANGs.

Your suggestion is the poster child of the old mantra "if the only tool you have is a hammer you tend to see every problem as a nail".

Re: Cracking down on research fraud

#32
post #4

Basic set of checks-and-balances: * Preregistration and adoption of open science practices * Public access to research results, methods, and data, with some exceptions (such as PII) * Federally-funded universities can't use NDAs or non-disparage agreements * Federally-funded universities must respond to records requests under terms similar to FOIA (note that FOIA has requestor pay costs) * Federally-funded universiti…

> The latter is tricky, since you don't want to smear the researcher without proof, but you also don't want to trust results.

IMO it's the build-up of univestaged allegations which makes them so damning. If we actually had justice the upper classes, small allegations were frequent, and investigations were de rigueur, we could actually have functioning "innocence until proven guilty" in the culture.

Relatedly, we should probably all have minor criminal records and it should be no big deal.

Re: Cracking down on research fraud

#33
post #20

Earlier quoted context omitted.

That's some guts to do the right thing. Respect.

I was not faced with such a shitty situation, and I probably would have done the same as well, but the right thing would have been to report it.

If you report it then the grant money is gone and everybody in the lab loses their job and all the PhD students are derailed.

Re: Cracking down on research fraud

#34

I think one of the big things that you can do is split up the data gathering and data analysis. Kind of like we don’t trust the companies to audit themselves, instead we have an outside firm. In this model, a researcher would create a hypothesis and collect the data. Their team would write the background and methods sections of the paper. Then the entirety of raw data would be sent off to a third party for data analy…

You would just get people massaging or generating the data to fit a conclusion prior to sending it out for statistical analysis. If people want to cheat they can find a way. The only way to really do what you are getting at is to construct core labs or CROs to run all experiments on behalf of the investigators. This is not infeasible in many cases (and is already done in narrow ways) but it requires hiring staff scie…

This.

And also actually fund researchers who do reproducibility work. May be even fund specialized teams that do only reproducibility work.

Re: Cracking down on research fraud

#35
Estimating from my own reading, 1% of research is fraud and 80% is worthless for other reasons. Numbers vary between fields.

If that's the case, what's the argument for why we should spend time doing something about the 1%? Solving 100% of the 1% wouldn't change the overall situation much.

Possible arguments include:

- Fixing the 80% is hard, but fixing the 1% is satisfying (to the aggressively conventional-minded, at least.)

- The 1% is wrong in a more harmful way than the 80%. Perhaps falsifying data is worse than hand-waving conclusions.

So if the maximum upside is 1% of wrong research removed, and the downside is quenching some fraction of the good 19%, it's probably better to leave it alone.

Re: Cracking down on research fraud

#36
post #11

The root of the problem is that we allow people with high esteemed credentials to run society. They jumped through a hoop. "You got straight As at 14 - 18 years old and got into an ivy league school as a result? Here run this venture fund." "You got a PhD in Economics with a good publication by P-hacking your secret data? Here take a run at the FED with power over the US Economy. Your Phd shows that you are the man f…

Publications really only look impressive to outsiders and gra d students.

I beg to differ. Publications are impressive because they are, in the very least, an indicator of the volume of work that a research group does in a domain. Writing a paper requires time and effort and, more importantly, focused work. As anyone who works and ever worked on academia is able to tell you, finding time to do anything of value, whether is exploring a new idea or continue working on any of the ideas you might have floating around, is the biggest challenge.

Re: Cracking down on research fraud

#37
Certain groups have been serious the replication crisis for 10-15y, but academic culture at large is simply not cut out to discuss fraud in a 'street epistemology' sort of way, such as the way security researchers might discuss cybercrime.

There's a wild amount of pushback to any amount of meta-criticism. But once you get past that point, many roadblocks remain.

In particular, there's extreme bias for meta-statistical methodologies that infer QRPs over other methods of investigation. Often, these methods aren't strictly necessary in context, and afford the opportunity to turn the metasci discourse into endless bikeshedding about the meta-framework rather than the object of dispute.

Many interested parties will participate/perform in this discourse, but few will sit down and really look at things as simple as the logical structure of the paper's claims, or even simpler problems with its content.

In the case of social psych, for instance, many problems lie with (a) stimuli and (b) unexamined assumptions on the part of the researchers that a certain manipulation holds, so these are important to assess. But extending critique to these things is seen as "reviewer 2" behavior — uncollegial, unfair, sniping, waaah etc.

Since "researcher x produces bad research, but evidence of fraud is only circumstantial" isn't sufficient grounds for doing much of anything, little comes of these efforts. When an investigation does nail a fraudster to the wall, well, so? The papers remain, the poisoned citation tree remains, the culture remains.

More than anything, the pushback in the form of tone-policing is what gets me. "Methodological terrorism" this, "reviewer 2" that. It shows how far from consequences the gatekeepers are. If you're a young person, whole branches of the academy have been pre-bankrupted for you. There's no hope there, short of a research path that manages to avoid citing any prior literature. But don't get tetchy with the grantlords!

Meanwhile, all around the US, real effects of this dogshit excuse for scientific inquiry can be seen every single day. Police departments are adopting new policies around known-bad implicit bias papers. These won't work. We know they won't work. We've known this for years.

What would make it stop? Every time cops kill an unarmed person, academics who still haven't retracted their implicit bias papers get fined?

There's the rub. You can't do much to force the point. Effective, timely measures would be fairly brutal ones, and academics aren't ready to admit this to themselves. In some ways, COVID-19 may end up being one of those measures, though it will disproportionately affect younger researchers.

Re: Cracking down on research fraud

#38

Earlier quoted context omitted.

I was not faced with such a shitty situation, and I probably would have done the same as well, but the right thing would have been to report it.

If you report it then the grant money is gone and everybody in the lab loses their job and all the PhD students are derailed.

That’s the fault of the person falsifying data, not the person reporting the fraud.

Re: Cracking down on research fraud

#39
I'd argue that a lot of what the authors describe isn't actually "fraud", it's more exploitation of an intentionally broken system.

I've definitely had authors on my papers who didn't do work. I've definitely written papers for people who didn't do work. I've definitely done peer reviews on behalf of PIs. Why do people do this? Because the regulators allow it and they want the system that way. Why should who wrote the paper have any impact on review? Why should it matter who the journal editors are? Why should it matter where the paper is from? Etc...

Re: Cracking down on research fraud

#40
I think there is outright fraud and subtle fraud.

For instance, most mouse studies specifically as it relates to aging, drug safety, and cancer research should be thrown out. This is well covered here:

https://m.youtube.com/watch?v=pRCzZp1J0v0

That’s not the only issue, but it’s a known issue, everyone is ignoring as it relates to all studies with the most common mice

Post reply on HN