Time to assume that health research is fraudulent until proven otherwise? • Hacker News Board

Re: Time to assume that health research is fraudulent until proven otherwise?

#21

Post by function_seven » Mon, Jul 19, 2021, 6:16 PM UTC

east2west wrote: post #14

Already reached the conclusion when I saw a genetic researcher presenting his p-value < 10^-40 as better than < 10^-10. I kept my mouth shut because I didn't want to ruin the poor guy's moment in the sun, but I knew it was time to get out.

My naive understanding is that "smaller p-value" == "more likely result is true".

I know there's always more nuance in statistical reasoning, but the first number is vastly smaller than the second one, right? Is it just that both are hilariously tiny and not credible? Or is there no additional value after you get into the one-in-billions territory?

Re: Time to assume that health research is fraudulent until proven otherwise?

#22

Post by ta988 » Mon, Jul 19, 2021, 6:16 PM UTC

When you want to do a proper work, your grants and papers get rejected because they are not innovatove enough or don't go far enough. So it is not a surprise that people that lied in their applications about what they can realistically do also lie when it comes to reporting results. Unfortunately there is no way out. I stopped counting how many reviewers of my grants disagreed on what was proposed, one saying that it was not innovative, the other saying that is was too risky to use this approach. We have a big problem in science, peer-review is broken and everything relies on it. And many reviewers are way out of touch about what happens in their field, I see reviews that clearly show the reviewer was sleeping for the last 10 years.

Re: Time to assume that health research is fraudulent until proven otherwise?

#23

Post by A4ET8a8uTh0 » Mon, Jul 19, 2021, 6:18 PM UTC

0xbadc0de5 wrote: post #4

Wouldn't this be the default scientific position? Or to take some of the hyperbole out of the statement, rephrase as: "is it time to assume that health research hypotheses are incorrect until a preponderance of data and reproducible studies prove otherwise?"

Scientific.. yes. But the studies are intended and/or used for a specific business purpose. The moment you recognize this simple reality, it becomes extremely difficult to take anything at face value. My wife is on the other side of the spectrum. She explicitly believes that companies/researchers/people generally want to do the right thing. It is infuriating, because my personal approach is my approach to games: 'shit, until proven otherwise'.

Re: Time to assume that health research is fraudulent until proven otherwise?

#24

Post by caseysoftware » Mon, Jul 19, 2021, 6:18 PM UTC

peytn wrote: post #2

No, but you need to be able to ask people who’re “in the know,” e.g. > When Roberts contacted one of the journals the editor responded that “I wouldn’t trust the data.” People already know what the bullshit results are.

If the editor didn't trust the data, why did they publish?

The people who keep informed on their field (aka "in the know") would then be tainted because they would likely believe a journal would vet the data, process, and researchers before publication.

Unless you mean "in the know" in that they know the entire publishing system is a scam... and the ripples from that are huge.

Re: Time to assume that health research is fraudulent until proven otherwise?

#25

Post by guscost » Mon, Jul 19, 2021, 6:19 PM UTC

hanniabu wrote: post #11

ChuckMcM wrote: post #6

Earlier quoted context omitted.

Exactly, that's the thing about science, nobody believes it until everything that has been tried to show it is wrong fails. I wish this were taught more in schools.

Yeah but you also can't go test 100 previous studies that the work you want to do is based on before you can even start yours. That's extremely inefficient, wasteful, and will tremendously slow progress.

> That's extremely inefficient, wasteful, and will tremendously slow progress.

Democracy is also “extremely inefficient, wasteful, and slow to progress”. A process with those deficiencies can still be the best available approach (although your example is perhaps too far in that direction).

Re: Time to assume that health research is fraudulent until proven otherwise?

#26

Post by zucker42 » Mon, Jul 19, 2021, 6:19 PM UTC

adolph wrote: post #18

Is this some vaccine hesitancy in disguise from BMJ? Stephen Lock, my predecessor as editor of The BMJ, became worried about research fraud in the 1980s, but people thought his concerns eccentric. Research authorities insisted that fraud was rare, didn’t matter because science was self-correcting, and that no patients had suffered because of scientific fraud. All those reasons for not taking research fraud seriously…

What? How do you figure that? Some person points out actual instances of poor quality research and you imply he is anti-vaccine? I can find no record of this person discouraging vaccination.

Re: Time to assume that health research is fraudulent until proven otherwise?

#27

Post by jerf » Mon, Jul 19, 2021, 6:20 PM UTC

ta988 wrote: post #22

When you want to do a proper work, your grants and papers get rejected because they are not innovatove enough or don't go far enough. So it is not a surprise that people that lied in their applications about what they can realistically do also lie when it comes to reporting results. Unfortunately there is no way out. I stopped counting how many reviewers of my grants disagreed on what was proposed, one saying that it…

Worse yet, it compounds. The people approving grants, seeing all these amazing results promised, will then raise the bar for what kind of results you're promising. Which means the next batch of promises will need to be that much more extreme to get approved. It's a race to the top... or the bottom, depending on your point of view.

Re: Time to assume that health research is fraudulent until proven otherwise?

#28

Post by ethanbond » Mon, Jul 19, 2021, 6:20 PM UTC

ta988 wrote: post #22

When you want to do a proper work, your grants and papers get rejected because they are not innovatove enough or don't go far enough. So it is not a surprise that people that lied in their applications about what they can realistically do also lie when it comes to reporting results. Unfortunately there is no way out. I stopped counting how many reviewers of my grants disagreed on what was proposed, one saying that it…

Yep. The incentives in science are all wrong. To maximize your chances of publication (i.e. keeping your job), you have to make the most outlandish claims you can possibly maybe defend. Additionally, the complexity of data/analysis is increasing every day while also the esoteric domain knowledge required to make any progress is deeper and more specialized.

Re: Time to assume that health research is fraudulent until proven otherwise?

#29

Post by hirundo » Mon, Jul 19, 2021, 6:21 PM UTC

I doubt most people understand the intensity of the incentive to mess with the data. In college I was a lab assistant for a professor who taught courses on research integrity and how to evaluate the quality of scientific papers. After thousands of hours of work on a study with routine 20 hour days collecting data, he wasn't getting what he needed to publish. At the tail end of one of those days I caught him with the equivalent of his thumb on the scale. He gave an excuse that he would have failed as an answer on one of his own tests. I argued a bit but then shut up. I kept shut up while that data was not excluded from the analysis that was eventually published. It wasn't enough to change the result, but still bothered me.

So yeah, trust maybe but verify definitely. The rewards for faking it are just too great for an honor system to be reliable.

Re: Time to assume that health research is fraudulent until proven otherwise?

#30

Post by long_time_gone » Mon, Jul 19, 2021, 6:21 PM UTC

onlyrealcuzzo wrote: post #7

gotoeleven wrote: post #3

The war on standards is creating a low trust society.

I don't know. The amount of people that take one poorly measured data point and use that as a north star seems more troubling to me - than a society with low trust. I'm not convinced a low-trust society is inherently bad. I am convinced that a society that has high trust in garbage data is bad.

==I'm not convinced a low-trust society is inherently bad. I am convinced that a society that has high trust in garbage data is bad. ==

It feels like we currently have both at the same time. There is "low trust" in any data that doesn't confirm an existing bias, but "high trust" in the data we want to be true.

I see it all the time in HN comments. Any study that contradicts the consensus is met with comments about "correlation =/= causation", while anything that supports the census is supported and correlation/causation isn't mentioned.