Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How many medical studies are faked or flawed?

web.archive.org

Post reply on HN
Print view View original

201–210 of 257 posts

Re: How many medical studies are faked or flawed?

#201

Post by eru »

junofan wrote: post #195
strken wrote: post #177

Earlier quoted context omitted.

"Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversaria…

I mean I’m as vaccinated as anyone, but the originally reported 95% effectiveness against symptomatic COVID-19 with the monovalent vaccine just doesn’t seem realistic in hindsight. Who knows what complex set of incentives was at play. If you’re running a trial site and your business relies on continued contracts from Pfizer, are you reporting every cough, fever, or headache?

I wonder if the 95% effectiveness was actually true: against the strains prevalent at the time they ran the trial. But effectiveness against later strains was diminished.

Re: How many medical studies are faked or flawed?

#202

Post by eru »

jcranmer wrote: post #182
ancorevard wrote: post #114

Earlier quoted context omitted.

"And that's when I believe people do have a somewhat best effort to maximize profits." Nope. I actually think that if you do scientific research as a company (profit) it may make you less bad/less likely to do fraud compared to academia (non-profit). Reason is that there are more ways to punish you, employees, board, investors, etc in a profit seeking vehicle, and as a profit seeking vehicle being caught must be part…

I think it's true that industry is more likely to produce reliable research compared to academia, but for a different reason. In academia, you essentially have the student who does the work, the professor, and the person who funded the grant, and that's essentially the sum total of people supervising the data. There's not a lot of people to call you out on fudged (or outright faked) data, and all of them are likely t…

Also the guys in the product team are interested in whether they can actually reproduce eg your novel synthesis method for your favourite molecule on a large scale.

The 'product' of academic research is a published paper. The product of industrial research is an actual product.

(This does not apply when your research is about eg effectiveness of a new drug. The product people can sell that drug on the strength of that research. Whether that research replicates or not is only of indirect concern in that case.)

Re: How many medical studies are faked or flawed?

#203

Post by eru »

metta2uall wrote: post #169
ancorevard wrote: post #114

Earlier quoted context omitted.

"And that's when I believe people do have a somewhat best effort to maximize profits." Nope. I actually think that if you do scientific research as a company (profit) it may make you less bad/less likely to do fraud compared to academia (non-profit). Reason is that there are more ways to punish you, employees, board, investors, etc in a profit seeking vehicle, and as a profit seeking vehicle being caught must be part…

But work in social sciences will often be inherently harder to replicate due to, for example, groups of human beings being more complex than groups of proteins - so there are a lot more variables, confounding factors, difficulties doing double-blind randomised trials, and so on...

Yes, but that's why the companies like Google or Facebook spend so much time and effort on their social science experiments.

Granted, they are mostly interested in a very small sliver of social science: 'how can you get people to directly or indirectly spend more time online and look at more ads'; but they are very, very interested in getting robust results that replicate well. They are also interested in figuring out how the results vary between different cultures and over time.

Re: How many medical studies are faked or flawed?

#204

Post by eru »

efitz wrote: post #168
ancorevard wrote: post #114

Earlier quoted context omitted.

"And that's when I believe people do have a somewhat best effort to maximize profits." Nope. I actually think that if you do scientific research as a company (profit) it may make you less bad/less likely to do fraud compared to academia (non-profit). Reason is that there are more ways to punish you, employees, board, investors, etc in a profit seeking vehicle, and as a profit seeking vehicle being caught must be part…

Exactly this. So many people have the opinion that private research must be flawed because of the profit motive, but the profit motive ensures that someone will be motivated to take oversight seriously, and have the power to punish misbehavior. Free markets, as ugly as they are sometimes, are still the best way we have of ensuring that incentives align with outcomes.

You might like https://gwern.net/backstop

War also works, to a certain extent, as a source of truth to align incentives with outcomes. But it's horribly expensive even in terms of economics alone, not to mention the human tragedy.

Luckily free markets work as a backstop, too. Add in free movement of people (who often want to come to better run places), and free movement of capital, and you have a winning combination.

Another stroke of luck: even if you only implement a very partial version of 'free', you still get partial benefits. Slightly freer markets are typically slightly more efficient. It's not an all or nothing proposition.

Re: How many medical studies are faked or flawed?

#205

Post by eru »

cutemonster wrote: post #142
autoexec wrote: post #124

Earlier quoted context omitted.

> Objectivity and honesty can be hard to find if all someone cares about is their reputation as a competent researcher or climbing the ladder. It seems like destroying the reputation and career of people who fake science would be a great start. If you're willing to fake data and lie to get results, there will always be an industry who'd love to hire you no matter how tarnished your reputation is. We need a better mea…

> destroying the reputation and career of people who fake science would be a great start When so many reports are faulty and fraudulent, that might instead be the great start of destroying the careers of those who would have revealed the fraudulent research? I wonder what'd happen if researchers got compensated and funding based on other things, unrelated to papers published. But what would that be

> I wonder what'd happen if researchers got compensated and funding based on other things, unrelated to papers published. But what would that be

See what they do in the parts of industry where they need their research to work.

Eg how do battery manufacturers compensate and incentives their researchers that are aiming to improve various characteristics of batteries? How do steel mills manage and reward their metallurgists? How does Intel's research work?

Re: How many medical studies are faked or flawed?

#206

Post by mistermann »

strken wrote: post #177
avalys wrote: post #173

Earlier quoted context omitted.

Ah, but don’t worry folks, those vaccines which generated $40B in profits for Pfizer and Moderna, those are just A-ok. Trust the science, which in this case is perfect, unassailable and uncorruptible (unlike everything else the pharma industry does). Don’t question the faith.

"Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversaria…

> "Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me.

What does "aren't outright bullshit" translate into in quantitative (%) terms?

Re: How many medical studies are faked or flawed?

#207

Post by a123b456c »

some_random wrote: post #18

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.

The scientific community ignored the replication crisis?

Dude. The scientific community created the replication crisis.

We are not ignoring it, I promise you. The reaction is nuanced by no one who matters is ignoring it.

Re: How many medical studies are faked or flawed?

#208

Post by BurningFrog »

autoexec wrote: post #132
BurningFrog wrote: post #63

Earlier quoted context omitted.

Doing research has a lottery element. You explore something that might reveal an important discovery. And sometimes it just doesn't. That doesn't mean you're a bad scientist, just an unlucky one. But it does mean you can't get tenure. So it's easy to understand why people fake results to secure a career.

> That doesn't mean you're a bad scientist, just an unlucky one. But it does mean you can't get tenure. That sounds like a really easy problem to solve. Just treat valid science as important regardless of the results. The results shouldn't matter unless they've been replicated and verified anyway.

It's a really easy solution to describe.

Actually implementing it across the academic world seems much harder.

Re: How many medical studies are faked or flawed?

#209

Post by mkl »

jltsiren wrote: post #111
colechristensen wrote: post #60

Earlier quoted context omitted.

If a university was actually in the business of seeking truth, there is much they could do to solve this. * working groups "red teams" whose whole job it is to find weaknesses in papers published at the university * post mortems after finding papers with serious flaws exposing the problem and coming up with constructive corrective actions * funding / forcing researchers to devote a certain amount of time to replicati…

Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. That's because universities are in the business of teaching. Apart from a few rare exceptions, universities don't have the money to hire redundant people. Instead of hiring many experts in the same topic, they prefer hiring a wider range of expertise, in…

> Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it.

Statisticians can find faults in many studies. Universities even employ statisticians to act as consultants for research in other departments, as that raises the quality of the institution's publications.

Re: How many medical studies are faked or flawed?

#210

Post by downWidOutaFite »

junofan wrote: post #195
strken wrote: post #177

Earlier quoted context omitted.

"Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversaria…

I mean I’m as vaccinated as anyone, but the originally reported 95% effectiveness against symptomatic COVID-19 with the monovalent vaccine just doesn’t seem realistic in hindsight. Who knows what complex set of incentives was at play. If you’re running a trial site and your business relies on continued contracts from Pfizer, are you reporting every cough, fever, or headache?

I don't think the claim was 95% against symptomatic covid, it was against hospitalization or death.

The vaccine was tested against the original virus but Omicron was almost a different disease with much more immune escape. We're lucky the vaccine held up as well as it did. This is also why the alternative doctors that wanted everyone to catch covid to build up herd immunity were wrong, but nobody seems to bring them up. These discussions just devolve into the usual polarized political talking points.

Post reply on HN