41–50 of 102 posts
Earlier quoted context omitted.
Explain that to the layman in a way the entire Internet will understand. Until you can ... they are for all intents and purposes.
Hardly equivalent "for all intents and purposes." The purposes of the layman are nothing like those of the research community.
So yes... public posted information should be layman ready
Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.
Explain that to the layman in a way the entire Internet will understand. Until you can ... they are for all intents and purposes.
Clinical trials are not basic research, though they rely on a great body of basic research, and are themselves experiments. Requiring that clinical studies have clearly defined end points serves the purpose of ensuring that the results are robust and well understood. This is important because the end goal of a clinical trial is developing a treatment of some sort that will go into many humans, and mistakes can be very harmful.
Choosing an appropriate endpoint for a clinical trial can also be very challenging. Say for example you're trying to advance a drug candidate for treating cardiovascular disease. You have many choices for how to measure that - chest pain, resting heartrate, cholesterol etc. It's very much possible that your drug does improve cardiovascular health, but your trial can fail because you chose the wrong endpoint.
When I perform an experiment, I do so with a hypothesis which is sometimes proven right, and sometimes proven wrong. Either way, the results can be interesting and form pieces of the large puzzle that is a scientific study. Further, questions that are addressed in a scientific study are generally going to be more broad than those addressed in a clinical trial - we don't really do exploratory clinical trials, but exploratory research is a very important part of science. For these sorts of studies, it is often difficult or impossible to have a well defined endpoint such as is required in a clinical trial. One example for you: I hypothesize that enzyme X has function Y. During my studies, I discover that enzyme X actually has function W! If the evidence I present for enzyme X having function W is solid, my results can still be great and useful, even if my initial hypothesis was wrong.
> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…
Analysis after the fact isn't useless. However the bar of significance needs to be much higher. You need a much larger sample size, or better yet design a new study (controlling for things the original didn't) to draw conclusions.
Note, technically statistics do not expect to find 5 false positives in 100. It is close enough for discussion and makes intuitive sense. If you want the real truth be prepared for a lot of math.
While some have pointed out that this very article could be considered post-hoc data mining, I think it is not. This exact kind of affect was what was intended by the requirement to pre-register. Looking afterwards to see if the intended affect (change in how many studies report benefits) appears to have happened in reality, makes perfect sense. They didn't institute the requirement at random, and then later consider…
> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…
[1] https://en.wikipedia.org/wiki/Multiple_comparisons_problem
Earlier quoted context omitted.
I would say that not all research needs to be held to the pre-registering standard of clinical trials. Some research is more exploratory, and should be interpreted that way.
That is right, And wrong. There are explicit exploratory methods you can use to carry out research and that is very fine, but most research today is done using an inference model, where you formulate a hypothesis and use data to corroborate it. That is one of the premises of big data that most people do not get: It enables you to do research explicitly exploratory again, while still being rigorous and somehow falsifi…
Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.
Explain that to the layman in a way the entire Internet will understand. Until you can ... they are for all intents and purposes.
"A reasonable takeaway from this article is that any one scientific study may contain flaws, or show bias, and therefore there is a possibility for it to be confirmed, improved, or disproven in the future. As a result, it is better to take early results with a grain of salt until additional supporting evidence is found, than to take them as gospel.
"It should be noted that these results are due to flaws in the scientific establishment, flaws in the human application of science, and flaws in the humans themselves, but not the scientific method itself. The scientific method itself is sound and can be trusted. You use it every day to see whether the water in your shower is too hot or cold, or whether it is raining outside."
> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…
The simplest, most straightforward explanation for the mechanism, and why it's bad, is this:
In the old scheme of things, the researchers would have found and reported "significant" bad outcomes connected to green jelly beans. With required preregistration, they must preregister 20 studies, so if 19 similar studies show no effect, and 1 of them shows "significant" effect, then it's likely that you're just dealing with random fluke.
> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…
Are you familiar with the multiple comparisons problem[1]? The problem is that in any sufficiently rich dataset, you can find something unusual-looking, and if you're not straightforward about how much digging you had to do to find it, it will look much more special than it actually is. So if a benefit is found by chance, that's fine, but it should only motivate further research which specifically look for that effec…
Indeed. But this doesn't disqualify looking for "something" as a valid and useful method of research to be a stage of the whole research chain. That ought to be allowed although research papers produced this way should make this clear.