Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cracking down on research fraud

undark.org

Post reply on HN
Print view View original

61–70 of 227 posts

Re: Cracking down on research fraud

#61

Post by rumanator »

ktaylora wrote: post #53
rumanator wrote: post #31

Earlier quoted context omitted.

> Industry kinda already has. Products launched. This suggestion is so detached from both the problem and the very nature of academia that it's straight-out laughable. I mean, what do product launches have to do building knowledge on the state of the art, identifying a novel idea, doing the iterative work to refine the idea, and finally document it to the public? Product launches at best require you to manage people…

Spoken like someone toiling away on their third postdoc position.

Ad-hominem aside, do you have anything to add to the discussion? I have been in academia for quite a few years, but I've been in the industry for longer than that. I state the point I stated because I indeed know first-hand what researchers do and what engineers and product managers do. What do you have to add to the discussion?

Re: Cracking down on research fraud

#62

Post by opportune »

ameliaquining wrote: post #57
tlb wrote: post #35

Estimating from my own reading, 1% of research is fraud and 80% is worthless for other reasons. Numbers vary between fields. If that's the case, what's the argument for why we should spend time doing something about the 1%? Solving 100% of the 1% wouldn't change the overall situation much. Possible arguments include: - Fixing the 80% is hard, but fixing the 1% is satisfying (to the aggressively conventional-minded, a…

The author is arguing for an expansive definition of fraud that includes things like p-hacking. Probably a lot more than 1% of research is p-hacked; that general category of problem might even account for a majority of your 80%.

I'm guessing throwing out "bad data" happens in a lot more than 1% of cases as well.

Re: Cracking down on research fraud

#63

Post by lrnStats »

DanielleMolloy wrote: post #30

I've read that before 2003, the whole of humanity has published as many scientific papers as from 2003 to 2016. So what happened around 2000? Who has turned the scientific mission into a blind competition for superficial metrics? So many people in science I meet (apart from the few who benefited from this system, and therefore were selected by it) are frustrated by publishing for the sake of publishing (not science)…

My guess is the education bubble. Loans are subsidized, universities have bigger budgets and want more studies done.

Re: Cracking down on research fraud

#64

Post by neffy »

logicslave wrote: post #51
opportune wrote: post #27

Earlier quoted context omitted.

All three of those examples are missing a step of like 5-20 years in the middle. Nobody is running a venture fund as a 22 year old ivy grad (barring someone who's taking over for their family member, but in that case it has nothing to do with being an ivy grad), nobody is running the FED right out of a PhD, nobody is a director at a big corporation right out of a PhD. Presumably once those people are in such high-pow…

Part of the point is that the gap in the middle often doesn't matter. I've hung around the Stanford crowd and as I enter my thirties I am awestruck by the opportunities they have regardless of what they have done since graduation. The credential is almost all that matters, as long as they dont mess up massively. A common path is: Graduate Stanford => Work at Mckinsey => Get hired into VC. Graduate Stanford => Raise 1…

That might have more to do with the subject than the individual.

Re: Cracking down on research fraud

#65

Post by lrnStats »

Similar to research fraud, I want the medical field examined for anti-science fraud.

I caught my Physician recommending an expensive and dangerous surgery that could be done by a dentist or surgeon. I asked if there was data, she said yes. There was no data. And the trend was using lasers rather than surgery since it's safer. I confronted her and she said-

"If you ask a physician, they will recommend a physician. If you ask a dentist, they will recommend a dentist."

This physician used factionalism rather than science.

I imagine this has happened on a massive scale.

Re: Cracking down on research fraud

#66

Post by ketzo »

tlb wrote: post #35

Estimating from my own reading, 1% of research is fraud and 80% is worthless for other reasons. Numbers vary between fields. If that's the case, what's the argument for why we should spend time doing something about the 1%? Solving 100% of the 1% wouldn't change the overall situation much. Possible arguments include: - Fixing the 80% is hard, but fixing the 1% is satisfying (to the aggressively conventional-minded, a…

I think the disproportionate harm argument is a big deal. A single high-profile case of fraud undermines trust in the entire field, and that’s critical to maintain.

Re: Cracking down on research fraud

#67

Post by aspenmayer »

opportune wrote: post #27
logicslave wrote: post #6

The root of the problem is that we allow people with high esteemed credentials to run society. They jumped through a hoop. "You got straight As at 14 - 18 years old and got into an ivy league school as a result? Here run this venture fund." "You got a PhD in Economics with a good publication by P-hacking your secret data? Here take a run at the FED with power over the US Economy. Your Phd shows that you are the man f…

All three of those examples are missing a step of like 5-20 years in the middle. Nobody is running a venture fund as a 22 year old ivy grad (barring someone who's taking over for their family member, but in that case it has nothing to do with being an ivy grad), nobody is running the FED right out of a PhD, nobody is a director at a big corporation right out of a PhD. Presumably once those people are in such high-pow…

That some aspects of the self-fulfilling nepotistic bureaucracies are meritocratic begs the question if those meritocratic elements only exist to justify the nepotism, and indirectly, the gatekeeping meritocracy.

Re: Cracking down on research fraud

#68

Post by MattGaiser »

OminousWeapons wrote: post #39

I'd argue that a lot of what the authors describe isn't actually "fraud", it's more exploitation of an intentionally broken system. I've definitely had authors on my papers who didn't do work. I've definitely written papers for people who didn't do work. I've definitely done peer reviews on behalf of PIs. Why do people do this? Because the regulators allow it and they want the system that way. Why should who wrote th…

Because we use heuristics instead of actual analysis.

How many people citing a paper or reading it actually have time to think that deeply about it?

The alternative to a world of trust and heuristics is a world where we are all bogged down trying to make decisions.

This is heavily demonstrated in recruiting. Resume reading is about 7 seconds a person. How long would it take if they spent a minute for each?

Re: Cracking down on research fraud

#69

Post by jjoonathan »

ktaylora wrote: post #10
Something1234 wrote: post #8

Earlier quoted context omitted.

Thanks for pointing out the problem with credentials. Name a better indicator then for being competent.

Industry kinda already has. Products launched. Teams managed. A candidate's publication record is a conversation starter during interviews, but I feel like years of experience at a company working on launching a successful project is a much more valuable currency.

Those are even easier to hack. Here's the result of measuring by projects launched: https://killedbygoogle.com/

Re: Cracking down on research fraud

#70

Post by Gatsky »

Research fraud gets a lot of attention because it is so black and white. But it is a symptom of larger problems. One issue is that the pace of progress is slowing, and as a result incremental gains are more prominent. This is fertile ground for fraudsters, as they can produce results which are plausible enough while seeming to be an important contribution to the field. All fraud fits into this category. Nobody makes a new grand unified theory of everything which they know is bogus. That would be too much work for a start.

The other issue is the huge expansion in university size. Most of the fraud I've seen or heard about all happens in university research departments. This shows you the importance of their incentive structure. One can make things up and not only succeed, but do better than your competitors in this research setting, AND get a tenured position with life-long security. All competitive fields where achievement recieves external and highly persistent rewards suffer from this problem, whether it be sport and performance enhancing drugs or Ivy league univesrity admissions or even venture captial funding (Theranos).

The natural response is to ask for more regulation and structural change in how research is conducted eg pre-registration, different statistical standards etc. But this has the major disadvantage of making life harder for the honest people. It also requires the creation of some parallel work force to handle all the checking. Research is already so difficult. Paradoxical effects, where such measures increase fraud, are definitely possible.

There will never be zero fraud. The aim should be to change how research is done to make the experience more humane, train and mentor young scientists carefully and avoid perverse incentives. As far as I can tell, nobody has any idea how to do this. Instead they want to create investigatory bodies which will siphon off money that could be used for research, and then ruin lives pursuing some key performance indicator like N successful fraud cases per quarter. This experiment was already run in the USA with the Office of Research Integrity, and it failed. Malcolm Gladwell, who I am not a fan of in general, has a good podcast about it [1].

[1] http://revisionisthistory.com/episodes/28-the-imaginary-crim...

Post reply on HN