Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

241–250 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#241
post #238

Earlier quoted context omitted.

> Actually, reproducibility is not considered a necessity in science. By whom? I think you're using as a premise the very matter that is being discussed.

I would say that most scientists, at the top of their game, who have years of trusted publications behind them, should be given some latitude to publish discovery science without the requirement of reproducibility, so long as they share their results in detail, and accept postdocs who then take the resulting research materials to their new labs. I didn't always believe this, but after spending a bunch of time reading…

For those interested in learning more about excellent scientists whose work could not be reproduced:

Tom Cech and his grad students discovered that RNA can have enzymatic activity. They proved this (with excellent control experiments eliminating alternative hypotheses) and yet, the community completely denied this for years and reported failed replication, when in fact, the deniers were messing up their experiments because working with RNA was hard. Cech eventually won the Nobel Prize.

Stanley Prusiner: discovered that some diseases are caused by self-replicating proteins. Spent several decades running heroic experiments that nobody could replicate (because they're heroic) before finally some other groups managed to scrape together enough skilled postdocs. He won the Nobel Prize, too.

Barbara McClintock- my personal favorite scientist of all time. She was soundly criticized and isolated for reporting the existence of jumping genes (along with telomeres) and it also took decades for other groups to replicate her work (most of them for lack of interest). Eventually, she was awarded the Nobel Prize, but she also stopped publishing and sharing her work due to the extremely negative response to her extraordinary discoveries.

Mina Bissel went through a similar passage, ultimately becoming the world leader in ECM/cancer studies. She will likely win a Nobel Prize at some point, and I think we should learn to read her papers with a certain level of trust at this point, and expect that her competitors might actually manage to level up enough to be able to replicate her experiments.

Re: Most scientists 'can't replicate studies by their peers'

#242
post #58

Earlier quoted context omitted.

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

> Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication. A 'world view' column in Nature suggested the same things last week [1]; the author described a paper of theirs [2]: > Yes, visual evidence can be faked, but a few simple safeguards should be enough to prevent that. Take a typical experiment in my field: using a tank of flowing water t…

I don't think this is particularly practical.

Most experiments run for years (literally) and no one is going to record or archive, let alone watch, years of footage to confirm that one paper is legit.

A brief experiment showing the apparatus and the collection of a few data points might be helpful for understanding the paper, but I can't see using it to verify a non-trivial experiment.

Re: Most scientists 'can't replicate studies by their peers'

#243
post #69

Earlier quoted context omitted.

If you can still, edit/remove the presumption of falsification and your argument has merit. I too percieve group think in climate change narratives, while not necessarily disbelieving the data. We are wrecking our environment, exact detrimental effects are still uncertain.

I'm sorry, but given that we're gold fish looking out of the bowl from the inside; trying to predict the future from vague memories of the immediate past; we have no clue what's causing what in this mess. Any one who claims otherwise is an ignorant prostitute, doesn't matter if they're aware or not. There are huge monetary incentives to pushing this agenda, and plenty of bullying to get people to walk in line. Have a…

"Have a look around today, and most scientists seem to agree that we're rather heading into the next ice age."

What scientists are you talking about? Also, didn't you JUST claim "we have no clue"?

By the way, global warming is not the case of some weird uptick in data correlating with some other data being interpreted as causally linked. In the case of global warming, it's really common sense (if you're trained in physics): We understand the infrared spectrum of CO2, and that by absorbing and retransmitting far infrared, it acts to slow the radiative transport of heat. This can be shown in the laboratory. Now, normally you think of the atmosphere being in a steady state, as plants grow they absorb CO2 (converting it into their cellular structure), and as plants die and are digesting or decay, CO2 is ultimately generated. A small amount is buried, but the amount buried is small compared to the overall cycle in a typical year, such that the atmosphere is roughly in equilibrium over the near term, although over the very long term (hundreds of millions of years), CO2 levels have dropped. (By the way, stars generally increase in brightness as they age, so when the CO2 levels were much higher, the Sun was dimmer). This buried carbon becomes fossil fuel, such as coal. Humans dig it up and burn it, but we've really only got good at this process on a large scale within the last 100-150 years (and even in the last 75 years, we've improved productivity by roughly an order of magnitude such that it takes a tenth as much manpower nowadays).

People about a century ago realized that the rate at which we were digging up this long-buried carbon was faster than the rate it was absorbed and buried naturally (makes sense, as the coal was produced over hundreds of millions of years), thus causing the CO2 level of the Earth to start to increase and thus the greenhouse effect to increase. It was just a side note at the time, a whimsical thought about the far future.

But today, the CO2 level has already dramatically changed since the 1800s, and we've also noticed that hey, we can see that predicted temperature rise as well, faster than would be explained just by coming out of the last ice age. This is a totally unsurprising finding if you know the infrared spectrum of CO2 and the rate at which fossil fuel is produced and burned (8-10 cubic kilometers of coal, 3.5 cubic kilometers of oil, about 4000 cubic kilometers of natural gas every year). You can reproduce the change in CO2 level over that time if you assume that roughly half of the carbon we burn every year is absorbed (by ocean, rocks, etc) and the other half stays permanently in the atmosphere.

That CO2 produces an insulating effect on the atmosphere is an indisputable fact. Earth would be much colder without this fact, and you can test it in the laboratory (and this effect is why Venus is hotter than Mercury, even though Mercury is much closer to the Sun).

That humans produce a very large amount of CO2 (i.e. significant fraction of atmosphere's total CO2 over decades) by burning long-buried fuels is an indisputable fact.

That the CO2 level has increased dramatically over the last 100 years is an indisputable fact.

The conclusion is that there MUST be some level of warming from human activity even before you look at the temperature data (which ALSO indisputably shows significant, off-trend warming over the last century), although the exact amount you'd expect depends strongly on the details of our climate system, such as feedback effects (both positive, i.e. increasing the effect, and negative, i.e. stabilizing or counter-acting the effect) from clouds, ice cover, vegetation changes, etc. We DIDN'T have to merely /guess/ at the causality direction after the fact once we saw that all the CO2 was not being fully reabsorbed, as it's a direct physical consequence of the infrared spectrum of CO2, something we can measure in the lab and even replicate from first principles quantum mechanics if we really felt like it. The fact that we observe warming is, to me, just the final validation of what we already knew would happen if we pumped a bunch of CO2 in the air.

By the way, I laugh at the idea that scientists (who can study anything and get grants one way or the other... and are pretty darned poor compared to similarly trained colleagues in the oil and gas business) have a "huge monetary incentive" to push this "agenda" but that somehow, corporations with tens of trillions of dollars worth of fossil fuel assets on the line don't have a similar agenda... I mean, the difference in financial incentive is absolutely absurd!

Re: Most scientists 'can't replicate studies by their peers'

#244

Earlier quoted context omitted.

That text is describing this situation: Context: You are a young scientist, under constant evaluation that may not only fire you, but also invalidate lots of your previous work. You have to produce flashy results to grow up on your career. You _really_ do not desire to get stuck where you are now. Now, let's decide on a new experiment to proceed. Do you choose: a - Boring important experiment that you can't hype but…

This quandary is a great argument for never becoming an academic researcher. It would seem to me that science needs to eliminate "career bias" or "mortgage bias" or "ramen bias" from its results. Negative results need to be just as publishable as significant results. If I were tyrant of the Ivory Tower, I would decree that results be blinded until after acceptance for publication. I would further decree that prestige…

Prestige ... points?

Re: Most scientists 'can't replicate studies by their peers'

#245
post #180

Earlier quoted context omitted.

Well, the burden of a policy has to be judged relative to what you're trying to accomplish. If the policy is ensuring that you achieve your ostensible goals, then that burden is justified. Based on your description, it actually sounds like it's making you do things exactly the way science is supposed to work! You quickly identify issues of "real effect or experimenter error or undocumented protocol?" -- and you preve…

The proposal has a problem in that it will increase the quality of results but will enormously slow down progress. E.g., looking from the perspective of a scientist-reader, if someone has spent a few months doing the ad hoc hands on experiments and achieved interesting observations, then I would want them to publish that research now, instead of spending another half a year to make the repeatable procedure or possibl…

Then maybe the problem is that the public is expecting results to be actually true, exacerbated by "hey what peer-reviewed literature are you supporting your argument with". If peer-reviewed literature is just a scratchpad for ad hoc ideas that may turn into something legit later, then the current standard is good enough, and we shouldn't be worrying that most of them are false.

OTOH, it's a problem if people are basing real-world decisions on stuff that hasn't reached textbook level certainty. That's pretty much what happened with dietary advice and sugars. "Two scratchpads say a high-carb low-fat diet is good? Okay, then plaster it all over the public schools."

Re: Most scientists 'can't replicate studies by their peers'

#246

You can back the Center for Open Science, which runs these reproducibility projects here https://cos.io/about/our-sponsors/

Seconding the support for COS. When they test reproducibility, they actually pair labs that are proficient in the particular technique with the original authors. The two labs (the originator and reproducer) work together to try to replicate the findings in the new environment. It's a smart way to work around the problem of labs trying (and failing) new techniques.

Re: Most scientists 'can't replicate studies by their peers'

#247

Earlier quoted context omitted.

> Unfortunately, the fact that this is occurring so rampantly will bolster anti-intellectuals and give them a very potent argument to point to when presented with facts. I agree with your point, but I suspect your choice of wording tells a little more of the story. You said "facts", but given your skepticism of their veracity, I think "claims" or "propositions" is more apropos. Consider the rhetorical difference of t…

Very good points. My own bias is to believe the claims of the scientific community and is thoroughly revealed in my language. Perhaps I'm the one being bilked.

Thanks. I think the discussion gets a little tricky because the groups "scientific facts" and "anti-intellectuals" are pretty broad categories.

I'm guessing that in a group of 1000 people that you would call "anti-intellectuals", some portion of them really do deserve that label.

Similarly, in the group of propositions you'd call "scientific 'facts'", some really are beyond reasonable dispute (i.e., Newton's laws in everyday-life settings). But that there are some other propositions which the academic community and their mouthpieces (New York Times, etc.) hold with unjustified confidence.

Re: Most scientists 'can't replicate studies by their peers'

#248
post #234

Earlier quoted context omitted.

I'm not a mathematician, but I had a math prof tell me that most publications only contain proof sketches, not full proofs. Can someone in the field comment?

I work as an applied mathematician. In general, I would say that this is incorrect. Virtually all of the papers that I read I would contend have full proofs. That said, I can sympathize with the sentiment in a certain sense. Just because a paper contains a proof doesn't mean that the proof is correct nor that it's comprehensible. Further, even if a paper went through peer review, it doesn't mean that it was actually…

Thanks for the insight!

Re: Most scientists 'can't replicate studies by their peers'

#249
post #111

Earlier quoted context omitted.

That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...

That's an absurd degree of cynicism. I just finished my PhD in chemical engineering, and I did the reverse of this. In fact, I successfully migrated the younger members of my research group to a completely open-source software stack for chemical simulations, so that in principle anyone with a Linux box could reproduce our results. We published all our code and plain text versions of the data. I'm not saying you're wr…

It differs depending on the field. Biology has a reputation for being extremely competitive. I've heard of PIs telling their students what they're forbidden to reveal at conferences for fear of getting scooped. As a CS grad student it was a completely different story: I was delighted just to get someone who was willing to listen to me talk.

Re: Most scientists 'can't replicate studies by their peers'

#250
Are there projects (companies, organizations, something else) trying to solve the problem or improve the situation? Maybe focus on a specific field as I'm sure there is a lot of domains where reproduction is just too expensive or too complex.

Reproducibility is a very important part of the scientific method, I would love to contribute to or work on that kind of project.

Post reply on HN