Too many medical trials move the goalposts. A new initiative aims to change that • Hacker News Board

Re: Too many medical trials move the goalposts. A new initiative aims to change that

#11

Post by StavrosK » Sat, Mar 26, 2016, 4:05 PM UTC

napoleoncomplex wrote: post #6

StavrosK wrote: post #4

Isn't what AllTrials[1] is doing a much better way of fixing medical science? [1]: http://www.alltrials.net/find-out-more/all-trials/

Both of these projects are from Ben Goldacre (not singularly, but he has a big part in both). COMPare is a continuation as far as I understand it, because now that all trials are registered and should follow those standards, abuse is still happening in the way COMPare points out.

Are all trials registered? Was AllTrials that successful? That would be extremely good news.

Re: Too many medical trials move the goalposts. A new initiative aims to change that

#12

Post by yummyfajitas » Sat, Mar 26, 2016, 5:19 PM UTC

nonbel wrote: post #2

We will never be able to enforce this and do productive research. It is all due to mathematicians' fantasy about how research works and researchers' fantasy about how stats work. Something always needs to be adjusted that is figured out during the study. Focus on testing precise predictions and observing similar results in different studies rather than significant differences. Despite it being much his "fault", Fishe…

Sure we can. Wall St does their best to enforce this because the alternative is losing money. Models work great in testing but fail horribly in the market.

Academic scientists feel that they can't be "productive" with rigorous stats because they consider "deploy the model in published paper form" rather than "see if it makes money" to be their end state.

Re: Too many medical trials move the goalposts. A new initiative aims to change that

#13

Post by nonbel » Sun, Mar 27, 2016, 3:20 PM UTC

napoleoncomplex wrote: post #5

nonbel wrote: post #2

We will never be able to enforce this and do productive research. It is all due to mathematicians' fantasy about how research works and researchers' fantasy about how stats work. Something always needs to be adjusted that is figured out during the study. Focus on testing precise predictions and observing similar results in different studies rather than significant differences. Despite it being much his "fault", Fishe…

The standard that COMPare is aiming to achieve, as far as I understand it, is the following: 1. Outcomes that will be measured for clinical trials need to be registered and public ahead of the trial (this is already achieved with recent regulation, as mentioned in the article) 2. If planned outcomes change, the change needs to be reported and explained, with new outcomes being again clearly stated. I really struggle…

Misrepresenting your methods is just fraud, so you will get no argument from me there.

However, think about the difference between A) setting a point prediction of your model as the null hypothesis vs B) "two groups are the same and samples were independent, etc" as the null hypothesis.

In the first case there is a very small range of plausible outcomes consistent with the researcher's theory. In the second case there is a very large range (usually 50%). In the case of scenario A, messing up the experiment (either on purpose, or accident) makes it more difficult to claim evidence consistent with your model. In the case of scenario B, the same thing makes this easier.

So researchers using scenario A are incentivized to be as careful and account for as many different sources of error as possible. Under scenario B, the incentive is the opposite, the sloppier the study the easier it is to get results consistent with the researcher's theory.