Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

Post reply on HN
Print view View original

231–240 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#231

Post by boznz »

misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

Repetition is given all the facts and original conditions the same or similar result can be expected.

Physics has its own cases such as cold fusion which is unreproducable and should not be if all the above hold true.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#232

Post by shkkmo »

bordercases wrote: post #222
batbomb wrote: post #176

Earlier quoted context omitted.

In fact, CMS and ATLAS are sort of same experiment repeated twice, just in parallel rather than serially. Repeatable doesn't imply identical experimental design.

Does it not? If the experimental design is not preserved, don't you open yourself to inconsistent controls between the experiments? Where do you draw the line?

If you think about it, complete replication of any experiment is useless, and impossible.

But that doesn't matter, because it is not the experiment you are trying to replicate, is the effect or the observation.

Carefully changing the design of the experiment can allow to verify that your explanation of the effect or observation is accurate.

Of course, if the change the experiment too much, a failure to replicated will have less useful information.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#233

Post by odbol_ »

Obi_Juan_Kenobi wrote: post #197
odbol_ wrote: post #186

Earlier quoted context omitted.

This is why I'm always skeptical of people zealously claiming "all GMOs are perfectly safe! It's been proven !" Yeah, proven by expensive studies that were funded by the company making the GMO. Who is going to pay for another study to try to disprove it? There's a reason we sprayed DET all over our vegetables for years before it was banned: there was no scientific studies proving that it was harmful, even though it c…

Do you mean DDT? Anyway, beyond the 'philosophy of science' issue of whether you can prove something, there is good affirmative evidence that existing GMOs are safe for numerous reasons. First, there's no mechanistic reason to think they would be dangerous. T-DNAs are not somehow magically toxic to humans; everything you eat is riddled with millennia of t-dnas, old viruses, transposon blooms, etc. etc. The technologi…

> First, there's no mechanistic reason to think they would be dangerous.

That's like saying "there's no reason to think peanuts would be dangerous" since humans eat them all the time. And yet, they are deadly to some humans. No one knows why.

But they do know that food allergies are far more prevalent in the U.S. than in other countries. And now, suddenly, people in Africa and China are starting to exhibit food allergies that the U.S. has had for a while. So what have we started shipping over to them that's causing these allergies? Who will fund that study?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#234

Post by fabian2k »

mmierz wrote: post #120

I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research p…

It's a bit redundant now, but it's a point that deserves repetition. Replication happens by the scientists that want to build on the original work and advance it. Not because they simply want to replicate it, but simply to be sure that you understand the method and all the practical details that are necessary.

If there is only a single paper about the result, most scientists will regard that with a certain suspicion. That doesn't mean we expect every new thing to be fake, but there are just too many ways these experiments can go wrong.

One problem is certainly that the negative reproduction of experiments are more often communicated informally, and not published. If you're active in the field you will likely hear from the scientists that tried to reproduce the results without any success. But most of these are not published.

Any experiment worth replicating, is usually also worth expanding and building on, and that is where the actual verification is taking place. Science works in this regard, but it doesn't work as cleanly and efficiently as one could hope for.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#235

Post by habitue »

A big distinction here is that different fields have different levels of dependence on prior results. In fields like psychology etc, you don't need the previous results to work in order to run your own experiment. In other words, if you cite a well-known paper saying "people seem to work faster near the color red" and your paper runs an experiment to see if they work faster near the color yellow, if the red paper is later unreplicable, it doesn't change the outcome of your experiment in any way.

In contrast, if you are in machine learning and you are extending an existing architecture you are very directly dependent on that original technique being useful. If it doesn't "replicate" the effectiveness of the original paper, you're going to find out quickly. Same for algorithms research. Some other comments here have mentioned life sciences being the same.

So I think there's a qualitative difference between sciences where we understand things in a mostly statistical way (sociology, psychology, medical studies) where the mechanism is unknown (because it's very very complicated), but we use the process of science mechanistically to convince ourselves of effectiveness. e.g. I don't know why this color makes people work faster/ this drug increases rat longevity / complex human interactions adhere to this simple equation, but the p value is right, so we think it's true. Versus sciences where we have a good grasp of the underlying model and that model is backed up by many papers with evidence behind it, and we can make very specific predictions from that model and be confident of correctness.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#236

Post by return0 »

throwaway729 wrote: post #103
return0 wrote: post #79

Earlier quoted context omitted.

We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences…

> we need to foster a culture that a) accepts repetition Do we? I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use…

Agreed , only important studies are worth replicating, the problem is that no major journal will publish a replication. Journals in general push for novelty rather than quality here.

b) can be a problem with meta-analyses and reviews. When gathering data "from the literature" , not all the data gathered is of the same quality/certainty, which can have a compounding effect. Or when someone from a mathematical or computational field tries to create a model using data reported in the literature. It is often difficult when working in an interdisciplinary environment to assess the quality of everything you read, especially if you re not familiar with all the experimental methods.

Also, off topic, but i wonder why you chose a throwaway account to weigh into this. i hope it's not a "science politics" reason.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#237

Post by jerf »

dalke wrote: post #209
jerf wrote: post #180

Earlier quoted context omitted.

I tend to agree, actually. In some sense the real solution is take science off its pedestal, as it does not generally deserve to be up there. The 17th-19th centuries were in some sense a fluke of low-hanging fruit, and the science of most of the 20th and the 21st centuries do not deserve to be regarded with the same worshipful gaze, a word I choose carefully. By taking it off its pedestal and subjecting it to a lot m…

"Science" is no longer on a pedestal. The PR campaigns against conclusions for leaded gas, smoking, acid rain, global warming, and vaccine safety, and the scientific development of leaded gas, ozone-depleting CFCs, Agent Orange/dioxin, etc., plus concerns like GMOs and Monsanto, mobile phone safety, plasticizers/hormone disruptors, and more make for a decidedly mixed view of science by the general public. As you can…

I think it's on a pedestal where it matters, where the funding decisions are being made. The public's opinion only matters in the long term. Though, admittedly, the long term is probably coming up pretty quickly. There's a lot of things that have had their opinion trending negative for so long that people have become blase about the negative trends suddenly coming due this year.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#238

Post by jerf »

bbctol wrote: post #217
jerf wrote: post #185

Earlier quoted context omitted.

Yes. But replication is a starting point. It isn't just about fraud, it's also about error and failure to communicate the experiment. (It's also beyond time for paper journals to just contain the relevant summaries and conclusions for given studies and for "what fits in a journal" to cease being "what fits in publication". Authors ought to be able to publish as much stuff as they want with a paper, including full sou…

So I work for a scientific journal (insert comments about how evil I am here) and we have much bigger problems with authors not providing full data than with us refusing it. It is almost always possible to upload a full data set and code to a paper; what we're working on now is standardizing formats as much as possible so it's easy to immediately see what equipment and reagents were used, where the experimental proce…

Yes, I do not mean to just criticize journals, authors also need to start realizing they need to produce that as well. 10-20ish pages of paper for the work of a dozen people for three years (and that for the good-to-great papers) is just not an acceptable payoff for the amount of money involved.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#239

Post by StanislavPetrov »

misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

>It's not as simple as that, for all sciences

Yes, it is - that's the very basis for science. One of the main problems is that many academic disciplines have been wrongly classified as hard science (see "social" sciences).

>The LHC has only been built once.. But everyone believes the results, and believes that they represent the Higgs.

Nobody intelligent "believes" anything. We examine evidence and draw tentative conclusions based on that evidence, always retaining doubt because, unless you are omniscient, there is always new information that can come to light that can cause you to change your conclusion. Science is a process. If you "believe" anything without doubt, you aren't a scientist, you are a priest.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#240

Post by logfromblammo »

dalke wrote: post #161
logfromblammo wrote: post #127

Earlier quoted context omitted.

No, it isn't. You calculate the fair market value of the public resources you used, and subtract what you paid the public for them. If it is positive, you have a publicly-supported project. So if you use that government-paid intern for several hours, you ought to pay their agency or department $7.25 for each. You pay for what you use, and there's no problem. If you work in a government-built facility, and you pay ren…

> "You calculate the fair market value of the public resources you used" Which is very hard for things with no market. I use government libraries. They are free to me. What is the fair market value of that? There are private and subscription libraries, so it's not like no market exists. What is the fair market value of time on Hubble? > "if you use that government-paid intern for several hours ... You pay for what yo…

The public does not want its funds diverted into private profits, period. Allowing any exceptions or leeway is a wide-open door to a trough filled with corruption and deceit.

You receive no special additional benefit from a library by being a researcher. Everyone can read the same materials as you do. Time on the Hubble costs more than any individual astronomer could pay. If you are keen on closed, private astronomy, you would need to check the NASA budget figures.

If you derive useful benefit from work done at your request, you need to pay the person doing it. If the intern is working for the government for no pay, how would they not just laugh in your face when you ask them to do work for you? You invented the hypothetical; I won't fix it for you.

If the military or state department could derive some benefit from subsidizing private research, they can bloody well do the research on their own. "US Army cures Dengue" would be great for both operations and PR, and would be a much better use of funds than a smart bomb that can stalk you on Facebook and blow up all your friends at the same time as you. If you as a private company want to sell a cure for Dengue on your own, then don't go begging the government for money. Fund it yourself!

Post reply on HN