Live data from Hacker News

How many medical studies are faked or flawed?

web.archive.org

251–257 of 257 posts

Re: How many medical studies are faked or flawed?

#251
post #205

Earlier quoted context omitted.

> I wonder what'd happen if researchers got compensated and funding based on other things, unrelated to papers published. But what would that be See what they do in the parts of industry where they need their research to work. Eg how do battery manufacturers compensate and incentives their researchers that are aiming to improve various characteristics of batteries? How do steel mills manage and reward their metallurg…

Aha, so they get paid the same, regardless of if their research works or not. And possibly some bonuses if things go well for the company ? And if they all the time do nothing useful and manipulate data, they get fired? But that's different from a PhD student -- they're not embedded in any organization that would notice if the research works or not? Maybe if the universities partnered somehow with different companies…

> Aha, so they get paid the same, regardless of if their research works or not.

I don't know. Is that speculation on your part, or something you figured out?

> But who would determine if a company had made use of a certain research paper? What would the company gain, by keeping track and reporting back? Maybe more good research

> (but I'd guess few companies would be that much forward-looking?)

That's why I am saying we should look what real companies are actually doing already in reality. We might have to leave our armchairs for that.

Re: How many medical studies are faked or flawed?

#252

Those are two wildly different things. Most things are flawed. Deliberately fabricating results is anathema to advancing science.

[flagged]

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: How many medical studies are faked or flawed?

#253

Earlier quoted context omitted.

You could try to build a science police but it won't work. Congress tried that in the 70s with the ORI. After a few failed prosecutions they basically gave up and are now forgotten - still there and they mount a few prosecutions a year but clearly it has no effect. The sheer scale of the problem cannot be fixed by trying to find people breaking "the rules" because there are no rules to begin with. Attempting to fix t…

You don't need a science police - allocating one half of all funding to replicating existing experiments will solve most of the problem. Why would one scientist commit fraud to protect the reputation of someone they've never met?

Oh boy. They do. They do it all the time. Too many of them see themselves as part of a "team", defending each other against outsiders. The Slack chats between virologists talking about the COVID lab leak hypothesis are a good example of this, they were very explicit about their aims there.

Huge funding for replication studies is commonly suggested, but it won't work. I'm basing that view on experience of having read and reported invalid papers across several different fields, spent years following this story and its various ins and outs, and have written extensively about the problems with pseudo-science coming out of official institutions.

Replication is the polite society way to talk about a whole range of problems. People say there's a replication crisis because it sounds a lot better than saying there's a large scale fraud and incompetence crisis. But it's not like there are papers out there that abstractly don't replicate, nobody could have known, and when someone tries and fails then there's suddenly a whole process that's followed to root cause why it failed and fix it. There isn't anything even close to that. What actually happens is that very obvious problems are ignored or kicked into the long grass as long as possible, often indefinitely, and sometimes scientists even argue they're being victimized by those reporting problems.

This article is a reasonable place to start, if you're new to the problem space:

https://fantasticanachronism.com/2020/09/11/whats-wrong-with...

It raises points that are often overlooked due to the "replication crisis" framing, points like:

• Many papers that can be replicated are only replicable because they're of little value (e.g. poor children have worse exam results). We don't want to be flooded with highly replicable science that tells us only what everyone already knew.

• Many papers are replicable but their claims are still invalid because their logic or methodology is wrong. This category covers significant amounts of COVID science, for example.

• It's often unclear exactly what the definition of replicable is.

But zooming out from replicability for a moment, I wrote an article with about some of the more egregious problems here:

https://blog.plan99.net/fake-science-part-i-7e9764571422

We're talking about a scientific culture that can't even stop obviously machine-generated text from being published on a massive scale. No other part of society has problems like this. Even talking about replication seems like a distraction whilst you have papers being published every day that contain obviously Photoshopped images, mathematically impossible numbers and AI generated text, yet nobody cares and getting even one case "resolved" (paper retracted) requires months or years of external pressure.

Think about how big the trust problems are with journalism. How to fix trust in journalism is talked about a lot on the conference circuit, but it's a hard problem. And yet newspapers don't publish garbled gibberish and faked images on a daily basis! That's the scale of the challenge faced with science reform.

And even if you somehow manage to drain that swamp, the first step before full replication is attempted is to get the original data, so the analysis steps can be replicated using the original numbers. Whole swathes of science fail here because academics refuse to reveal their data. This happens even when they've been made to sign a statement agreeing that they'll provide it on request. Again, their employers do nothing, so where's the pressure point? Only massive fiscal punishment could cause culture change here but the NSF has a single institutional goal of dishing out as much money as possible. To quote Alvaro de Menard,

Why is the Replication Markets project funded by the Department of Defense? If you look at the NSF's 2019 Performance Highlights, you'll find items such as "Foster a culture of inclusion through change management efforts" (Status: "Achieved") and "Inform applicants whether their proposals have been declined or recommended for funding in a timely manner" (Status: "Not Achieved"). Pusillanimous reports repeat tired clichés about "training", "transparency", and a "culture of openness" while downplaying the scale of the problem and ignoring the incentives. No serious actions have followed from their recommendations.

It's not that they're trying and failing—they appear to be completely oblivious. We're talking about an organization with an 8 billion dollar budget that is responsible for a huge part of social science funding, and they can't manage to inform people that their grant was declined! These are the people we must depend on to fix everything.

Re: How many medical studies are faked or flawed?

#254
post #251

Earlier quoted context omitted.

Aha, so they get paid the same, regardless of if their research works or not. And possibly some bonuses if things go well for the company ? And if they all the time do nothing useful and manipulate data, they get fired? But that's different from a PhD student -- they're not embedded in any organization that would notice if the research works or not? Maybe if the universities partnered somehow with different companies…

> Aha, so they get paid the same, regardless of if their research works or not. I don't know. Is that speculation on your part, or something you figured out? > But who would determine if a company had made use of a certain research paper? What would the company gain, by keeping track and reporting back? Maybe more good research > (but I'd guess few companies would be that much forward-looking?) That's why I am saying…

> Is that speculation on your part, or something you figured out

That's just normal monthly wages, how things usually work.

> look what real companies are actually doing already

But you can't look at what companies are doing now, to find out if new research is useful? The companies can't yet have started doing the things that any new & good research enables (since it wasn't known before).

Could take years until they make use of the research

(But maybe you meant something else)

Re: How many medical studies are faked or flawed?

#255
post #251

Earlier quoted context omitted.

> Aha, so they get paid the same, regardless of if their research works or not. I don't know. Is that speculation on your part, or something you figured out? > But who would determine if a company had made use of a certain research paper? What would the company gain, by keeping track and reporting back? Maybe more good research > (but I'd guess few companies would be that much forward-looking?) That's why I am saying…

> Is that speculation on your part, or something you figured out That's just normal monthly wages, how things usually work. > look what real companies are actually doing already But you can't look at what companies are doing now, to find out if new research is useful? The companies can't yet have started doing the things that any new & good research enables (since it wasn't known before). Could take years until they…

I mean look at what companies have been doing in the past to organise and incentivise research, and see what worked.

Look at what processes worked (and didn't work!). Not at what specific inventions worked.

Re: How many medical studies are faked or flawed?

#256
post #255

Earlier quoted context omitted.

> Is that speculation on your part, or something you figured out That's just normal monthly wages, how things usually work. > look what real companies are actually doing already But you can't look at what companies are doing now, to find out if new research is useful? The companies can't yet have started doing the things that any new & good research enables (since it wasn't known before). Could take years until they…

I mean look at what companies have been doing in the past to organise and incentivise research, and see what worked. Look at what processes worked (and didn't work!). Not at what specific inventions worked.

If you have any books or articles you like related to that, it'd be interesting to hear :-)

(Research by Google about psychological safety comes to my mind.)

Re: How many medical studies are faked or flawed?

#257

I heard somewhere 11% of cancer research can be replicated. Does anyone have numbers with sources for replicatability?

Out of 53 'landmark' oncology studies, only 6 were replicable in this study conducted by a biotech based in Thousand Oaks, California.

https://www.nature.com/articles/483531a

Post reply on HN