Live data from Hacker News

Time to assume that health research is fraudulent until proven otherwise?

blogs.bmj.com

51–60 of 491 posts

Re: Time to assume that health research is fraudulent until proven otherwise?

#51

Already reached the conclusion when I saw a genetic researcher presenting his p-value < 10^-40 as better than < 10^-10. I kept my mouth shut because I didn't want to ruin the poor guy's moment in the sun, but I knew it was time to get out.

My naive understanding is that "smaller p-value" == "more likely result is true". I know there's always more nuance in statistical reasoning, but the first number is vastly smaller than the second one, right? Is it just that both are hilariously tiny and not credible? Or is there no additional value after you get into the one-in-billions territory?

Yeah I'm also mystified by that comment. You are correct that smaller P is better. Those near physics level P-values are not totally unheard of for genetics either, because they have very large databanks with hundreds of thousands of data points in them and the ability to do large analyses over them, so they can obtain a lot of statistical power.

Re: Time to assume that health research is fraudulent until proven otherwise?

#52
post #18

Is this some vaccine hesitancy in disguise from BMJ? Stephen Lock, my predecessor as editor of The BMJ, became worried about research fraud in the 1980s, but people thought his concerns eccentric. Research authorities insisted that fraud was rare, didn’t matter because science was self-correcting, and that no patients had suffered because of scientific fraud. All those reasons for not taking research fraud seriously…

What? How do you figure that? Some person points out actual instances of poor quality research and you imply he is anti-vaccine? I can find no record of this person discouraging vaccination.

Ok, maybe its just a coincidence that it was published right when the current dangers of misinformation are so high and so many are failing to believe science.

Re: Time to assume that health research is fraudulent until proven otherwise?

#53
post #18

Is this some vaccine hesitancy in disguise from BMJ? Stephen Lock, my predecessor as editor of The BMJ, became worried about research fraud in the 1980s, but people thought his concerns eccentric. Research authorities insisted that fraud was rare, didn’t matter because science was self-correcting, and that no patients had suffered because of scientific fraud. All those reasons for not taking research fraud seriously…

This is almost an apples/oranges comparison.

There's research and then there's research.

Yes, all approved medicines are the result of medical research. But approving medicines isn't the entirety of medical research.

I'm sure you've heard of the adage: egos are large when the stakes are low. This is related to that. The push to "get published" is so great that if you know that your research has no practical effect or that it won't affect anyone, you can kind of make up whatever results you want. As long as it publishes. The goal isn't to find the truth or to answer a question, the goal is to get a byline in a paper. And you get more and better bylines by discovering something radical and novel rather than by saying "Nope, doesn't work, just like expected".

On the flip side, when the results really matter, you'll find people do proper due diligence. Especially when your results will be essentially confirmed practically by billions of people on the planet. When the stakes are high, we wind up being way more cautious.

Of course, I hear your meta-concern. Because, yes, people will use this paper to pull the "Science is a lying bitch"* card. But it is also an issue that must be dealt with or at the very least acknowledged. As the article itself notes, someone did notice it in the 80s, but due to very concern of casting doubt on medical research, they kind of just hoped it wouldn't be an issue. And now the issue is too great to deal with simply.

In the end, medical, and really all scientific, research cannot be "hit driven". Failure must be an option. And if this month's issue of The BMJ is a bit "boring" or "thin", then so be it. The focus must be more on finding the correct answer rather than the flashier answer. Even when the stakes are small.

*Science is a lying bitch: From the It's Always Sunny in Philadelphia episode "Reynolds vs. Reynolds: The Cereal Defense". One of the characters uses the fact that certain noted scientists had incomplete ideas about certain scientific phenomenon or weren't completely right on every subject as proof that science itself was flawed and couldn't be trusted on the subject of evolution and therefore one should believe the biblical account of creation because the bible hasn't been changed since it was written.

Re: Time to assume that health research is fraudulent until proven otherwise?

#54
post #35

Earlier quoted context omitted.

https://shop.powerinsole.com/en/blogs/news/doppelblindstudie... I actually bought one of the devices and took it apart on video. There is nothing inside except 4 magnets and a plastic printed card. No active circuits and no power source and no components such as resistors or capacitors and certainly there is no microchip inside. edit below are links to pictures of the device and accompanying text as originally on fac…

Thank you. What a load of infomercial buzzword nonsense. The last part about some “further research and experience” showing even better results if the product is worn all the time is especially galling.

But one should look into Darsch Scientific if you want to get deeper into how this is all organised. https://www.dartsch-scientific.com/en/ they have produced papers for powerinsole and are used by other companies in the magnetic magic industry to provide a veneer of scientific credibility. For example https://waveguard.com/en/studies/

Re: Time to assume that health research is fraudulent until proven otherwise?

#55
post #20

I recently had the privilege of trying to do the right thing when I identified fraudulent research carried out by an institution in Austria. The initial response of the institution was positive but when I pressed for further details on how such things could happen suddenly nobody anymore would take my call. The research was paid for by a private company to pimp a nonsense product. The research was never published in…

[deleted]

Re: Time to assume that health research is fraudulent until proven otherwise?

#56

Earlier quoted context omitted.

I guess if you don't even feel comfortable sharing the identity of any of the parties, we really have no chance to identify bad actors from good as a community.

"Naming and shaming" comes with risk - probably a lot more risk than upside. The risk is that the parties get something actionable, kind of like "probable cause" in criminal cases. The upside is maybe a small effect on a few HN minds that might remember this when considering this Uni's reputation. The other risk is that it's an act that can easily be abused. It is very easy to level charges against someone without pr…

[deleted]

Re: Time to assume that health research is fraudulent until proven otherwise?

#57

Already reached the conclusion when I saw a genetic researcher presenting his p-value < 10^-40 as better than < 10^-10. I kept my mouth shut because I didn't want to ruin the poor guy's moment in the sun, but I knew it was time to get out.

My naive understanding is that "smaller p-value" == "more likely result is true". I know there's always more nuance in statistical reasoning, but the first number is vastly smaller than the second one, right? Is it just that both are hilariously tiny and not credible? Or is there no additional value after you get into the one-in-billions territory?

I estimate the risk of human error (chose the wrong modeling assumptions, bug in data processing code, etc.) at least ~1%, so there really isn't any point in claiming any statistic that is smaller than that.

Re: Time to assume that health research is fraudulent until proven otherwise?

#58
It is about time to focus on the right problem: management standards that cause this crap to be pushed, and the effective immunity from consequences companies have when they lie.

Why blame scientists when power is actually with management?

And why let management and investors get away with this? It is about time "limited" liability had a pass through liability for this type of stuff: if you lie, knowingly or not, there are consequences and the consequences bypass limited liability. I bet if that happens this type of crap would immediately cease!

Re: Time to assume that health research is fraudulent until proven otherwise?

#59

Already reached the conclusion when I saw a genetic researcher presenting his p-value < 10^-40 as better than < 10^-10. I kept my mouth shut because I didn't want to ruin the poor guy's moment in the sun, but I knew it was time to get out.

My naive understanding is that "smaller p-value" == "more likely result is true". I know there's always more nuance in statistical reasoning, but the first number is vastly smaller than the second one, right? Is it just that both are hilariously tiny and not credible? Or is there no additional value after you get into the one-in-billions territory?

It would be, but such an imbalance of p-values is unrealistic. 10^-10 probability? If your probabilistic model includes even a one in a billion chance of messing up (10^-9), a p-value of 10^-10 is already too small. That’s before you look at 10^-40... so they are probably both wrong.

A nice demo of this effect is DNA matching in criminology. Although DNA matching of suspects to DNA samples can be insanely accurate, in practice it is limited by the incidence of monozygotic (identical) twins, which is about 3 in 1,000. You cannot be more certain than this that you got a match, essentially.

Re: Time to assume that health research is fraudulent until proven otherwise?

#60

Earlier quoted context omitted.

I don't know. The amount of people that take one poorly measured data point and use that as a north star seems more troubling to me - than a society with low trust. I'm not convinced a low-trust society is inherently bad. I am convinced that a society that has high trust in garbage data is bad.

==I'm not convinced a low-trust society is inherently bad. I am convinced that a society that has high trust in garbage data is bad. == It feels like we currently have both at the same time. There is "low trust" in any data that doesn't confirm an existing bias, but "high trust" in the data we want to be true. I see it all the time in HN comments. Any study that contradicts the consensus is met with comments about "c…

I think a better term for that is "brain damage". So what if all our brains are built damaged. Saying people are not reasoning machines, but social maneuvering machines doesn't change the fact they are kinda broken...
Post reply on HN