Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How many medical studies are faked or flawed?

web.archive.org

Post reply on HN
Print view View original

241–250 of 257 posts

Re: How many medical studies are faked or flawed?

#241

Post by whatshisface »

mike_hearn wrote: post #225
whatshisface wrote: post #40

Earlier quoted context omitted.

If the replication studies were done at a reasonable rate there would be no incentive to produce bogus results because you'd be caught before you could go through an entire career as a "successful scientist."

There is no being caught :( Who is going to do the catching, exactly? There's no science police. Universities don't seem to care about anything except plagiarism. Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to roc…

I think the most natural place for reform is from the people who are paying for the research, if they realized that what they are spending money for has no value if it's being made up. Although there are a lot of scientists who by the principle of self-interest seem willing to waste their life in exchange for a small salary it makes less sense why the NSF would take that offer.

Re: How many medical studies are faked or flawed?

#242

Post by tppiotrowski »

yellowcake0 wrote: post #146
some_random wrote: post #64

Earlier quoted context omitted.

There is always value in expanding the breadth and depth of human knowledge, even if it doesn't seem useful to you, right now. That of course assumes that knowledge is true, which is the crux of the problem now.

Unfortunately, the preponderance of very low value research in the literature puts a significant burden on the scientists who have to sift through a lot of garbage to find what they're looking for. Even if the work is ostensibly correct (much of it is not), it really doesn't do anyone much good, except for the authors of course. But now every undergraduate and every parent's vanity project at Andover wants a first au…

I think language models will help us sift through the garbage. I have a hard time finding answers using Google (too much SEO garbage), but ChatGPT does a pretty good job of filtering the noise. It's not perfect, but it saves lots of time vs. Google.

Re: How many medical studies are faked or flawed?

#243

Post by mharig »

fer wrote: post #237
mharig wrote: post #232

Earlier quoted context omitted.

You trust health authorities? Come on.

You on the other hand believe that, somehow, absolutely independent (and often confronted) authorities, both governmental and private, would magically agree on something false instead of calling each other out. That's the-moon-landing-was-fake tier denial-ism.

If the political pressure is big enough, yes.

You cannot compare the moon landing to the COVID circus.

Re: How many medical studies are faked or flawed?

#244

Post by jltsiren »

mkl wrote: post #209
jltsiren wrote: post #111

Earlier quoted context omitted.

Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. That's because universities are in the business of teaching. Apart from a few rare exceptions, universities don't have the money to hire redundant people. Instead of hiring many experts in the same topic, they prefer hiring a wider range of expertise, in…

> Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it. Statisticians can find faults in many studies. Universities even employ statisticians to act as consultants for research in other departments, as that raises the quality of the institution's publications.

Statisticians can find faults in many studies, but without a sufficient background in the topic, they can rarely tell if those faults actually matter. And the statisticians who have the background are usually already involved in the study anyway.

Re: How many medical studies are faked or flawed?

#245

Post by mike_hearn »

whatshisface wrote: post #241
mike_hearn wrote: post #225

Earlier quoted context omitted.

There is no being caught :( Who is going to do the catching, exactly? There's no science police. Universities don't seem to care about anything except plagiarism. Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to roc…

I think the most natural place for reform is from the people who are paying for the research, if they realized that what they are spending money for has no value if it's being made up. Although there are a lot of scientists who by the principle of self-interest seem willing to waste their life in exchange for a small salary it makes less sense why the NSF would take that offer.

But that can't happen at the moment. The core problem is that civil servants and politicians are easily bullied by claims they aren't experts and thus can't judge if "expert" work is any good or not. So they just dish out money to anyone who works at a university, expecting them to self-police, but they don't.

There are only two ways to fix that, as far as I can see:

1. Have highly technical, scientifically trained and skeptical politicians who police academia and ensure research money is well spent.

2. Stop governments funding research in the first place.

(1) just doesn't seem feasible. There are so many problems there. (2) is feasible. So I suspect you're going to start seeing calls from the right to defund universities over the next ten years or so. The underlying motivation may be that universities are strongholds of the left, but the stated justification will be the high levels of research fraud.

Re: How many medical studies are faked or flawed?

#246

Post by cutemonster »

eru wrote: post #205
cutemonster wrote: post #142

Earlier quoted context omitted.

> destroying the reputation and career of people who fake science would be a great start When so many reports are faulty and fraudulent, that might instead be the great start of destroying the careers of those who would have revealed the fraudulent research? I wonder what'd happen if researchers got compensated and funding based on other things, unrelated to papers published. But what would that be

> I wonder what'd happen if researchers got compensated and funding based on other things, unrelated to papers published. But what would that be See what they do in the parts of industry where they need their research to work. Eg how do battery manufacturers compensate and incentives their researchers that are aiming to improve various characteristics of batteries? How do steel mills manage and reward their metallurg…

Aha, so they get paid the same, regardless of if their research works or not. And possibly some bonuses if things go well for the company? And if they all the time do nothing useful and manipulate data, they get fired?

But that's different from a PhD student -- they're not embedded in any organization that would notice if the research works or not?

Maybe if the universities partnered somehow with different companies, and the researchers got extra compensation if a company decided to make real world use of the research?

(On top of some base salary)

But who would determine if a company had made use of a certain research paper? What would the company gain, by keeping track and reporting back? Maybe more good research

(but I'd guess few companies would be that much forward-looking?)

Re: How many medical studies are faked or flawed?

#247

Post by whatshisface »

mike_hearn wrote: post #245
whatshisface wrote: post #241

Earlier quoted context omitted.

I think the most natural place for reform is from the people who are paying for the research, if they realized that what they are spending money for has no value if it's being made up. Although there are a lot of scientists who by the principle of self-interest seem willing to waste their life in exchange for a small salary it makes less sense why the NSF would take that offer.

But that can't happen at the moment. The core problem is that civil servants and politicians are easily bullied by claims they aren't experts and thus can't judge if "expert" work is any good or not. So they just dish out money to anyone who works at a university, expecting them to self-police, but they don't. There are only two ways to fix that, as far as I can see: 1. Have highly technical, scientifically trained a…

Or working from historical analogies, 3) find scientists with a public and obvious track record of replication, and maybe even a few of the fraud hunters, and put them in charge of the NSF.

Re: How many medical studies are faked or flawed?

#248

Post by mike_hearn »

whatshisface wrote: post #247
mike_hearn wrote: post #245

Earlier quoted context omitted.

But that can't happen at the moment. The core problem is that civil servants and politicians are easily bullied by claims they aren't experts and thus can't judge if "expert" work is any good or not. So they just dish out money to anyone who works at a university, expecting them to self-police, but they don't. There are only two ways to fix that, as far as I can see: 1. Have highly technical, scientifically trained a…

Or working from historical analogies, 3) find scientists with a public and obvious track record of replication, and maybe even a few of the fraud hunters, and put them in charge of the NSF.

You could try to build a science police but it won't work. Congress tried that in the 70s with the ORI. After a few failed prosecutions they basically gave up and are now forgotten - still there and they mount a few prosecutions a year but clearly it has no effect. The sheer scale of the problem cannot be fixed by trying to find people breaking "the rules" because there are no rules to begin with.

Attempting to fix this within the framework of government would be incredibly distracting for the state, as it'd lead to endless fights and debates. For example, a reasonable person appointed to run the NSF might conclude they should just defund whole subfields of the social sciences, because there's no way to make them scientific at all, but good luck lasting in the job if you do that. A big part of the problem is that scientists are in hoc to a specific political ideology, so they will have powerful friends in Washington who want to see them stick around producing useful propaganda.

Re: How many medical studies are faked or flawed?

#249

Post by whatshisface »

mike_hearn wrote: post #248
whatshisface wrote: post #247

Earlier quoted context omitted.

Or working from historical analogies, 3) find scientists with a public and obvious track record of replication, and maybe even a few of the fraud hunters, and put them in charge of the NSF.

You could try to build a science police but it won't work. Congress tried that in the 70s with the ORI. After a few failed prosecutions they basically gave up and are now forgotten - still there and they mount a few prosecutions a year but clearly it has no effect. The sheer scale of the problem cannot be fixed by trying to find people breaking "the rules" because there are no rules to begin with. Attempting to fix t…

You don't need a science police - allocating one half of all funding to replicating existing experiments will solve most of the problem. Why would one scientist commit fraud to protect the reputation of someone they've never met?

Re: How many medical studies are faked or flawed?

#250

Post by gmd63 »

mattkrause wrote: post #239
gmd63 wrote: post #74

Earlier quoted context omitted.

I'm interested more in why it specifically would be hard to just show people what you are doing (or did) instead of telling them a story at the end of months or years. Critical details and oversights can be lost in translation between reality and LaTeX that could be easily pointed out by third party observers.

Who is going to watch thousands[0] of hours of video, most of which will show a scientist fiddling around with setscrews or hunting down a loose ground wire? And what scientist is going to be happy with constant surveillance that's intended to be shared (so no picking your nose, losing your temper, or making mistakes)? [0] Literally, thousands. I just finished a paper about a project that we started three years ago.…

That does sound like a lot of footage, but you wouldn't watch every second just as you likely haven't read every word of every scientific paper you've skimmed or referenced.

The footage is there to support inquiries into how you did your experiment, just as you would skim a youtube video about repairing an appliance to get to the part you need to see.

And yeah, it would have to be a new level of prestige backing this way of documenting your methods. Obviously people prefer being trusted, but p-hacking and replication problems have proven that we can't have nice things.

Post reply on HN