Why Most Published Research Findings Are False (2005)

41–42 of 42 posts

Re: Why Most Published Research Findings Are False (2005)

#41

Post by bmogen » Mon, Oct 01, 2018, 5:08 PM UTC

chriskanan wrote: post #7

majidazimi wrote: post #5

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…

I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication…

There is a huge difference between physical and computer sciences. I agree that it's an excellent use of time for new students if the main factor for the work is time/salary. This paper was aimed towards medical/bio/clinicians and when you add material costs, the variability of biology and multiple people required to run large experiments everything falls apart. One of my PhD projects (Primate neuroscience and new medical devices) took 4 years2 grad students2 staff6 animals (only 2 made the paper)animal housing costs for 4 years + ~$150k in materials. And there are only ~10 research labs in the world that have the ability to do this type of research so no one is going to . We see a lot more replication via extension - "this theory worked for this group, what if we take that as true and extend it in a new direction, where does the science fall apart there?" I'm not arguing that replication isn't important and a lot of false positives get through into the literature but without 10x-ing the research budget and infrastructure as well as changing the publication incentives there isn't going to be any real movement.

Re: Why Most Published Research Findings Are False (2005)

#42

Post by chrisfinazzo » Tue, Oct 02, 2018, 2:12 PM UTC

nonbel wrote: post #40

chrisfinazzo wrote: post #33

Earlier quoted context omitted.

If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong. @epistasis' using the term 'p-hack' is incredibly apt in this case. The ability for a study to be deemed "correc…

>"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?" Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science. Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion…

It's really the exceptional case I'm thinking of here.

Basically, you screw up such that the data you've collected is meaningless. Sure, you could do the statistical analysis correctly and get a result, but it ends up being the opposite not because the relationship is actually different, but because some part of your design is flawed. In that case, you would never end up with a correct result.

(This is of course the worst possible outcome, but I would be remiss if I didn't mention it here.)

Although I have no background in it, I suppose the same could happen in the other sciences if there are side effects that you don't anticipate and don't measure accurately enough.

^ This again is why reproducibility is important. Earlier in this thread, @majidazmi mentioned that a major focus of their thesis was to prove that a prior experiment was incorrect in its findings.

Post reply on HN