Too many medical trials move the goalposts. A new initiative aims to change that
1–10 of 13 posts
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#2We will never be able to enforce this and do productive research. It is all due to mathematicians' fantasy about how research works and researchers' fantasy about how stats work.
Something always needs to be adjusted that is figured out during the study. Focus on testing precise predictions and observing similar results in different studies rather than significant differences. Despite it being much his "fault", Fisher pointed this out nearly 60 years ago:
Fisher, R N (1958). "The Nature of Probability". Centennial Review 2: 261–274
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#3[deleted]
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#4Isn't what AllTrials[1] is doing a much better way of fixing medical science?
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#5We will never be able to enforce this and do productive research. It is all due to mathematicians' fantasy about how research works and researchers' fantasy about how stats work. Something always needs to be adjusted that is figured out during the study. Focus on testing precise predictions and observing similar results in different studies rather than significant differences. Despite it being much his "fault", Fishe…
The standard that COMPare is aiming to achieve, as far as I understand it, is the following:
1. Outcomes that will be measured for clinical trials need to be registered and public ahead of the trial (this is already achieved with recent regulation, as mentioned in the article)
2. If planned outcomes change, the change needs to be reported and explained, with new outcomes being again clearly stated.
I really struggle to see how enforcing this will prevent productive research. This is basic scientific hygiene, not fantasy. No one is arguing that studies shouldn't be adjusted, just that the adjustments should be clearly stated.
Completely changing the goals of a study after failing on your initial goals, and then claiming success without mentioning any of the failures along the way is poor science, because you're masking a lot of potentially significant findings along the way that others can benefit from as well.
Not to mention that these standards have been agreed to by all the largest journals decades ago, with big PR claims of how this is needed etc., yet are never followed in practice. The responses of some journals (NEJM, JAMA that I recall) have been shameful, and contradictory to standards they claim to follow.
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#6Isn't what AllTrials[1] is doing a much better way of fixing medical science? [1]: http://www.alltrials.net/find-out-more/all-trials/
Both of these projects are from Ben Goldacre (not singularly, but he has a big part in both).
COMPare is a continuation as far as I understand it, because now that all trials are registered and should follow those standards, abuse is still happening in the way COMPare points out.
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#7We will never be able to enforce this and do productive research. It is all due to mathematicians' fantasy about how research works and researchers' fantasy about how stats work. Something always needs to be adjusted that is figured out during the study. Focus on testing precise predictions and observing similar results in different studies rather than significant differences. Despite it being much his "fault", Fishe…
Do you think Study 329 was productive research? (Productive for patients, that is, as was productive for GSK's income.)
If a study is always adjusted then how can you make a precise prediction about it?
Once you've done the research and found a post-hoc correlation, what additional tests are needed before making a policy decision, like recommending Paxil for depressed youngsters?
If there was outcome switching, is it okay to omit noting that in the paper?
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#8Isn't what AllTrials[1] is doing a much better way of fixing medical science? [1]: http://www.alltrials.net/find-out-more/all-trials/
These are different goals.
All trials is about not hiding when you have done a trial that doesn't produce your desired result. (Publication Bias) COMPare is about being transparent if you change the purpose of your trial while it's happening. (Outcome switching)
Both problems can be countered by good preregistering practice. And both are important issues that need to be fixed. (And by the way: not only in medicine. This is a problem that is widely a problem in science.)
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#9We will never be able to enforce this and do productive research. It is all due to mathematicians' fantasy about how research works and researchers' fantasy about how stats work. Something always needs to be adjusted that is figured out during the study. Focus on testing precise predictions and observing similar results in different studies rather than significant differences. Despite it being much his "fault", Fishe…
It is true that in practice, you can't foresee everything that is going to happen in a practical experiment (you wouldn't do it anyway if it was the case). But it doesn't justify not enforcing clean research practice. If you need to move the goalposts, you can do it, it's call "adjusting the hypothesis". You make a new hypothesis and you test it properly. Sometimes you need to iterate a lot, and you fall into the risk of https://xkcd.com/882/. Which still underlines the importance of having a strict scientific approach.
Re: Too many medical trials move the goalposts. A new initiative aims to change that
#10Isn't what AllTrials[1] is doing a much better way of fixing medical science? [1]: http://www.alltrials.net/find-out-more/all-trials/
These are different goals. All trials is about not hiding when you have done a trial that doesn't produce your desired result. (Publication Bias) COMPare is about being transparent if you change the purpose of your trial while it's happening. (Outcome switching) Both problems can be countered by good preregistering practice. And both are important issues that need to be fixed. (And by the way: not only in medicine. T…
Oh hmm, I thought you had to pre-register your purpose with AllTrials as well. Too bad, they could have killed two birds with one stone.