Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

161–170 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#161

Post by jonlucc »

cwyers wrote: post #133
jonlucc wrote: post #129

Earlier quoted context omitted.

You can add to this that if you are in a facility with both rats and mice, you should never house them in the same room, and you should probably not enter a mouse room after being in a rat room. In the wild, rats are predators of mice.

But in the wild, there aren't separate rooms for predators and prey, either. If laboratory rodent experiments are so reliant on such synthetic conditions, what's the point?

You've gotten a couple of answers, but my answer might rely heavily on my field. I work in early stage pharma discovery research. So our goal isn't to determine the basal level of some cytokine, for example, in normal mice. Our goal is typically to see that cytokine's response to treatment, or stimulus, or stimulus then treatment. In other words, the mouse is a living system one step more complicated than a dish of cells, which is its value to us. Sometimes, once you take a drug to non-human primates or humans, you have to drastically change the type of study you run to determine efficacy.

Re: Most scientists 'can't replicate studies by their peers'

#162

Post by smallnamespace »

cwyers wrote: post #150
jpfed wrote: post #141

Earlier quoted context omitted.

If some mice are housed near rats and others are not (or simply vary in how far they are from the nearest rat), that will introduce variation into your measurements. For experiments not specifically about mouse/rat interactions, this variation is irrelevant noise that makes it harder to detect or characterize the effects the experimenters actually care about.

That's not my point, though. Like, if you conduct physics experiments in a vacuum because interactions between objects and atmosphere isn't part of what you're studying, that's fine. If everyone in physics is conducting experiments that way, then suddenly you're left with the question of, it turns out that in real life I encounter atmosphere all the time , and how much can these experimental results tell me about the…

Yes, because you are trying to control your variates? Just because physics is the most amenable to experimental control does not mean that the only real science is physics.

I mean by that measure, medicine is not a science either because we don't know most of the possible confounding variables. That doesn't mean that attempting to use the scientific method still isn't the correct choice.

Re: Most scientists 'can't replicate studies by their peers'

#163

Post by jonlucc »

SilasX wrote: post #120
JumpCrisscross wrote: post #40

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

Or require the author to only communicate to the technicians via the same means that will be published with the paper -- i.e. Make sure anyone reading the paper has the same spec for the experiment as those who performed the original. [1] That would help prevent such "hidden specs" from entering the experiment. [1] Note: this implies that authors cannot be part of the experiment or conduct it themselves, because they…

In my job, I work with an outside contract research organization. The way this works is pretty similar to what you're describing, and its a living hell. For starters, it takes 6-10 drafts back and forth to get a protocol to start with, and the resulting document is never shorter than 12 pages. Then, once they start a study, if we get any aberrant data, the question becomes "did they mess that up, or is that data real?". Of course anything we want to be sure about, we run twice, but when you have a panel of 20 compounds, you can't duplicate all of the work, so some compounds get dropped even if the data were not "real". Also, and maybe this would be different with another CRO, there are often very stressful conversations in which they are trying to avoid being blamed (because if they screwed up, we aren't supposed to pay them the full price), but we just really want to know what happened. Lastly, you can tell a lot by being hands-on with a study; there's a lot you can miss if you aren't in the room with the study. Just my 2 cents.

Re: Most scientists 'can't replicate studies by their peers'

#164

Post by cwyers »

IanCal wrote: post #137
cwyers wrote: post #112

Earlier quoted context omitted.

> Think about the epistemological ramifications of this. This this this this this a million times this. If your experiment with lab rats needs to be done this finely to be replicated, is it really telling you anything about the real world? Because of course, we don't really care about the behavior of laboratory rats, right? Not in proportion to how often we do studies with laboratory rats versus other species of anim…

The point, I think, is that you want to control for things that really alter the outcome. It's not that a clean cage means that the end-product no longer works, but you want to compare "didn't have drug" vs "did have drug" not "didn't have drug" vs "did have drug but the fire alarm went off this morning". Unless your effect size is much larger than all the noise, you could easily miss a true result. Worse, you could…

So, let's continue with your example of a drug being given to the rodents. Now, you want to control diet -- you don't want your control group eating differently than your treatment group. This ends up being relatively easy to do with laboratory rodents, they aren't popping down to McDonalds for lunch or lying in their food diaries, like human subjects might. You buy your rodent food from one rodent pellet supplier, and you don't change brands or SKU during the experiment, and voila, control.

And this works fine if the drug you're studying works identically given all rodent diets. But you don't know that it does! You're not controlling for the variable of diet in the sense that you know the effect of the variable of interest across all possible diets. You know how the drug works conditional on one specific rodent diet. And if you're not putting what brand of rodent food you use, and that causes replication failures, that suggests that you don't understand the effect of the drug as well as you thought you did. And if you have an entire field of research that undergoes such replication failures often, it's fair to start wondering how much of what gets called "science" isn't the study of the natural world but the study of the very specific sets of conditions that happen in labs in American and European universities.

Re: Most scientists 'can't replicate studies by their peers'

#165

Post by michaelmrose »

wycx wrote: post #149
michaelmrose wrote: post #115

Earlier quoted context omitted.

At that point its actually not even science and shouldn't be funded

The way you get funded is by being able to do things that others can't do, or by being better at some technique than everyone else. Giving away all your hard won tricks by putting all of them in a methods section takes away your advantage come funding time.

The problem is that it is far easier and cheaper to produce stuff that looks like science but isn't so if you fund stuff without being able to prove it/reproduce it then you will probably end up funding 90% bullshit while starving real science for money.

Any real scientists ought to recognize that secrecy as a strategy is terrible for science as a whole even if it is very temporarily good for them.

Re: Most scientists 'can't replicate studies by their peers'

#167

Post by thehardsphere »

BeetleB wrote: post #104
thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publishe…

>>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls.

>No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get published either. And no one writes grant applications that are for replication studies. The grant will likely go to someone else.

Sorry, let me be clear: If an interesting result is published, people will go to the trouble. Most results are of limited interest and mediocre.

Re: Most scientists 'can't replicate studies by their peers'

#168

Post by Declanomous »

jonlucc wrote: post #159
Declanomous wrote: post #142

Earlier quoted context omitted.

Yeah I did an experiment with metabolism in relation to temperature with rodents in college. Our results were just absolute garbage for the most part. I think two of us owned cats, walk in temperature controlled fridge was incredibly loud and the light never turned off, and the fridge was on the opposite side of the building from the only lab with the correct equipment to run the experiment. Plus the enclosure for me…

There's some neat software now (not sure if it was around when you were in college) to translate video into reliable data about the movement of animals. I've used it for fish, but the same software can be used to track people around a room or to track mice in a cage. I think that might be much more useful than, say, how many backflips a mouse does in their cage.

This was about 9 years ago, so it probably existed in a certain sense, but our lab computers were still apple II's because that was what the software for the VO2 sensors was written for. The lab group doing the caffeine experiment was trying to measure whether the resting VO2 went up when dosed with caffeine, so the fact that the rats were moving at all was the primary problem.

We tried using a machine vision program written by a grad student at another college to count trees from old survey photos at one point, and it did not work well at all in anything with more trees than a park-like setting. We ended up just having a human circle all the trees they saw and I wrote a program to detect the circles. It worked much better. The program I created was based on something similar used to count bacteria colonies on petri dishes.

Re: Most scientists 'can't replicate studies by their peers'

#169

Post by dragonwriter »

thehardsphere wrote: post #167
BeetleB wrote: post #104

Earlier quoted context omitted.

>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publishe…

>>That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. >No, no, a thousand times no! Most studies do not have follow up studies that confirm/refute the original. Often such a followup study is hard to publish. If you manage to reproduce it, you cannot publish unless it presents a new finding. If you fail to reproduce it, it often doesn't get publis…

> Sorry, let me be clear: If an interesting result is published, people will go to the trouble.

That's only true for a definition of "interesting" that is more like the sense most people assign to "astounding" or "groundbreaking", and even then it's not guaranteed, just somewhat probable. If it's both groundbreaking and controversial (in the sense of "immediately implausible to lots of people in the domain, but still managing to draw enough attention that it can't be casually ignored as crackpot"), like, say, cold fusion, sure, there will be people rushing to either duplicate or refute the results. But that's a rather far out extreme circumstance.

Re: Most scientists 'can't replicate studies by their peers'

#170

Post by yummyfajitas »

wycx wrote: post #111
hannob wrote: post #90

Earlier quoted context omitted.

> I was chatting with a colleague last summer, and asking > him for more details about a method he had published that > I was having trouble reproducing. After pointing me to a > few papers that didn't include the relevant details, > hefinally told me that "if we told everyone about how it > was done, anyone could do it." That's not science, that's bullshit. Can you please expose that? Scientists shouldn't simply get…

That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...

That's not how science works. During my scientific career I gave code and data away to any and all who wanted them. That includes critics, competitors, and anyone else who was curious.

I felt my results were solid enough (and that my skill at producing more results was good enough) that this wouldn't hurt me. This was just how things worked in my field (physics).

Interestingly, my experimental colleagues rarely felt that they couldn't reproduce results they saw in a paper.

Post reply on HN