Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

211–220 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#211
post #148

In that case, macro-economics is simply disqualified from being scientific. It's almost impossible to repeat large-scale events, controlling for all variables. Have to say I'm not particularly impressed with the quality of Nautilus's analysis.

> In that case, macro-economics is simply disqualified from being scientific.

Well, duh? (In other words: of course it's not a science.)

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#212
post #186
post #120

I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research p…

This is why I'm always skeptical of people zealously claiming "all GMOs are perfectly safe! It's been proven !" Yeah, proven by expensive studies that were funded by the company making the GMO. Who is going to pay for another study to try to disprove it? There's a reason we sprayed DET all over our vegetables for years before it was banned: there was no scientific studies proving that it was harmful, even though it c…

This is just a large strawman argument. Talking about the safety of GMOs is more like talking about the safety of "Medicine"- you're actually talking about the safety of a large variety of different products. Pushing the narrative that all GMOs should be treated equally is like pushing the narrative that all medicines should be treated equally, which is nonsensical.

The very idea that there can be "something wrong with GMOs" is as anti-scientific as the narrative that there can be "something wrong with medicines". It's anti-scientific on principle, because it's not the safety of individual products you are taking issue with, it's the very existence of the entire scientific field.

Anyways, it takes about 10 years for the average Ag product to make it to market. That's on par with the time for the average drug. If you are serious about safety, start talking about real improvements that could be made to the Ag approval process. Start talking about the flaws in individual studies. Until you do something like that, your anti-science blanket dismissal of an entire field is as ridiculous as someone who is "against medicines".

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#213

Earlier quoted context omitted.

I think you've got my argument completely backwards - I'm pointing out that if repetition as implied by the article is everything, then results from e.g. the LHC are discounted. Similarly, I am not saying that you don't need to repeat - just that it isn't the be all and end all of what defines 'science'. Supporting this interpretation is that I mentioned DAMA. Nobody is accusing them of not taking care, but nobody re…

There is a philosophical assumption underpining astronomy and cosmology, the so-called "cosmological principle."[0] This is an assumption (albeit not completely unfalsifiable) that physics across the universe at large scales is the same for all observers, that here on earth there should be similar to physics in andromeda. Of course , all scientific experiments done have been done on or near earth, and thus, our inter…

But we can measure gravity in faraway galaxies. We don't need to actually go there, we have telescopes :)

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#214
post #157

Earlier quoted context omitted.

> The entire point of science, and this is not a hyperbole, is that results are reproducible. No, it's not. The point of science -- its end -- is to understand the natural world. Or to cure diseases. Or, more cynically, to learn how build big bombs and more manipulative adverts. Reproducible results are the means, not the end. I know that seems like hair splitting, but it's important. Epistemological purity can do ju…

The only way we "understand" the natural world is by making verifiable predictions. If those predictions can't be consistently verified, then we don't understand the relevant phenomenon at all, and we haven't learned anything. > To be concrete about it, I'll always prefer a single (set of) instance(s) of a well-designed and expertly executed experiment over 10 reproductions of a crappy experiment. I'd take 2-3 repeti…

I think the issue here is that scientists want to get funding, have prestige, and perhaps learn about the world in the process. The public wants to be cured of diseases, see new technologies, and learn about the world too.

There is a disconnect between the motivation and capability of scientists in the current funding system and what the public wants. So an easy solution is that if the public wants reproduce-able science, they need to pay for it. I'm sure some scientists who couldn't make it into Harvard or Caltech (ie., me) and thus can't do cutting edge science would be happy to take the dollars, have a living, and just reproduce the work of others. But you can't simply declare to scientists they should do X while not enabling them to.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#215
post #120

I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research p…

>>If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result. Said scientist will gain immense fame and recognition if they show established ideas are wrong. (Source: academia Physicist)

You only get famous if you produce a new order that replaces the old one. Throwing stones at the old order doesn't actually get you very much unless you provide something better.

Especially in psych/bio, a negative finding can almost always be blamed on the experimentalist -- even more so if they are young. It is no great feat to conduct a study so poorly that fail to detect a signal others have reported.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#216

Earlier quoted context omitted.

The difference is maths. E.g. Einstein proved his theory of gravity mathematically, THEN it was proved via empirical observation. We don't have a way to mathematically prove a drug for depression a psychological phenomena.

What does "prove mathematically" mean here? He constructed a self-consistent mathematical theory, and validated that it reproduced both the special relativistic and Newtonian limits, but I don't think "proved mathematically" is a good way to state that. You can't "mathematically prove" a theory agrees with reality except by reconciling it with experiments (possibly indirectly through links to previous theories), so t…

Did you google it?

https://en.wikipedia.org/wiki/Mathematical_proof

You're missing the point of what I said w/ the Einstein example. He proved it via maths. Whether or not it was empirically validated is irrelevant. It was true mathematically

Today, this cannot be done with fields like medicine and psychology. No one can create a mathematical proof for a cure for cancer, make a treatment based upon that proof, and have it work on the first try

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#217
post #185
post #92

Earlier quoted context omitted.

But there should also be replication of the concepts of the study under a different paradigm, otherwise you might just be finding out effects of the particular procedure. An original study is rarely "wrong" (fraud is a serious problem, but not a typical one), it's that it's merely one data point, from one procedure, and a lot of the time the effect that can be replicated is "real," it's just a side effect of the proc…

Yes. But replication is a starting point. It isn't just about fraud, it's also about error and failure to communicate the experiment. (It's also beyond time for paper journals to just contain the relevant summaries and conclusions for given studies and for "what fits in a journal" to cease being "what fits in publication". Authors ought to be able to publish as much stuff as they want with a paper, including full sou…

So I work for a scientific journal (insert comments about how evil I am here) and we have much bigger problems with authors not providing full data than with us refusing it. It is almost always possible to upload a full data set and code to a paper; what we're working on now is standardizing formats as much as possible so it's easy to immediately see what equipment and reagents were used, where the experimental procedures came from, what data was gathered and when, and how exactly it was analyzed without grabbing tons of files in different formats. "As much information as authors want" is high for an excellent but small set of authors who are probably right that the journal should do more to facilitate hosting their data--for most authors, that's far less information than we actually need.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#218
post #21

My first reaction to this headline was "duh." Of course we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims! My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the…

" he is coming across people who treat science like religion."

The only difference between Scientists and people in other fields is that Scientists completely lack self-awareness.

Science is obviously systematically biased.

Have you ever been to a good school? Every word a prof says is designed to make you think he is smart. That's the only way they make careers. They 'live in the identity' of being smart.

It's laughable and hyper competitive.

Bad studies ensue.

It's obvious to anyone with a basic grasp of human behaviour.

The only thing that surprises me is how they are unwilling to admit the problem.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#219
post #169

Cue all the people justifying their pseudoscientific behavior. If it is too expensive to fund twice, it shouldn't be funded once. If that means the LHC and LIGO wouldn't get done, then we should have only funded one of them. We need to remain skeptical of those results until replicated by a new team. Even one replication is pretty weak... Independent replications of experiment (and the corresponding independent repor…

You do realize that scientists who work on the LHC have the highest repeatability standards of any science profession, right?

The LHC experiment is not the issue here.

There is a lot of transparency there, a lot of well meaning people with a lot of oversight.

I suggest most would admit 'there could be a problem' there, but it's out in the open if there is.

The problem of lack of repeatability I think has to do with subconscious bias on the part of the experimenters which will be less pronounced when there are 5000 people working on it.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#220

Earlier quoted context omitted.

ATLAS, CMS, ALICE, LHCb, etc. are all different experiments that just happen to share the same accelerator. They are done using different detectors, designed, built, operated and the data is analyzed by different groups of people. If you follow your logic, that the different experiments using the same accelerator negates the whole thing, to the extreme then doing two experiments on the same planet/solar system/univer…

I think you've got my argument completely backwards - I'm pointing out that if repetition as implied by the article is everything, then results from e.g. the LHC are discounted. Similarly, I am not saying that you don't need to repeat - just that it isn't the be all and end all of what defines 'science'. Supporting this interpretation is that I mentioned DAMA. Nobody is accusing them of not taking care, but nobody re…

[deleted]
Post reply on HN