Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

Post reply on HN
Print view View original

221–230 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#221

Post by vibrio »

refurb wrote: post #130
mmierz wrote: post #120

I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research p…

As a former life scientist, I completely agree. If something truly revolutionary is published and it's relevant to your own work, it's very common to repeat the experiment yourself. If it works, great, continue on with your own work and reference the original paper when you publish new results. If it doesn't work, the outcome can range from: (1) mentally note and move onto an alternative if they are available to (2)…

As a former life scientist, I completely agree also. Things do get repeated. If the findings aren't robust, scientists cut bait and move on. This knowledge is shared informally, for certian. This "Formalzation" of the experimental repetition would be difficult since no one wants grant money or their careers spent generating "yeah, s/he's right about that' outcomes.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#222

Post by bordercases »

batbomb wrote: post #176
misnome wrote: post #151

Earlier quoted context omitted.

I think you've got my argument completely backwards - I'm pointing out that if repetition as implied by the article is everything, then results from e.g. the LHC are discounted. Similarly, I am not saying that you don't need to repeat - just that it isn't the be all and end all of what defines 'science'. Supporting this interpretation is that I mentioned DAMA. Nobody is accusing them of not taking care, but nobody re…

In fact, CMS and ATLAS are sort of same experiment repeated twice, just in parallel rather than serially. Repeatable doesn't imply identical experimental design.

Does it not? If the experimental design is not preserved, don't you open yourself to inconsistent controls between the experiments? Where do you draw the line?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#223

Post by yk »

misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

Physics is a special case for several reasons, [1] most importantly there is a good theory with the effect that different experiments measure the same thing. So once you start looking for a specific deviation from the standard model, you can not only look at accelerator experiments, but also at g-2 experiments [2], or lamp shift or beta decay in the case of the electro-weak interaction. (Or at different places in accelerator experiments, in the sense that data at one point should lead to corrections at another point.)

In other disciplines, the role of repetition is a lot more important. The look elsewhere effect means, that your statistical significance depends on all other research, published or unpublished. [3] If you are doing a repetition study, the look elsewhere effect just goes away. So good science in the first place is very important of course, but there is a lot of value in doing repeated experiments for fundamental statistical reasons. Especially in fields were there is no good predictive theory (everywhere except physics) and where it is very hard to go to very high significance.

[1] I recently read

John Lewis Gaddis, The Landscape of History, 2004

which is a essay on epistemology of history and in a way tries to reject "physics envy," with the argument that physics is everything where there is a good theory, so in a way where we are winning. And consequently physics envy looks a lot like cargo cult, other disciplines need to figure out their own epistemology.

[2] Incredible high precision experiments which measure the magnetic moment of electrons.

[3] https://xkcd.com/882/

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#224

Post by analog31 »

zouhair wrote: post #204
analog31 wrote: post #174

Earlier quoted context omitted.

I suppose the funding agencies could play a role in this. When you submit a proposal that cites prior work as a basis or motivation for you project, you should be required to show that either 1) The cited works have been reproduced, or 2) you're going to reproduce them.

And also we need to start publishing negative studies way more.

I dimly recall some physics papers from years ago, with titles such as: "New experimental limit on the existence of..." Whenever my project was going particularly badly, we'd lighten things up by describing it as a negative study. One such highlight was: "New experimental limit on the existence of oil in the vacuum pump."

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#225

Post by johncolanduoni »

anotherhacker wrote: post #216
johncolanduoni wrote: post #189

Earlier quoted context omitted.

What does "prove mathematically" mean here? He constructed a self-consistent mathematical theory, and validated that it reproduced both the special relativistic and Newtonian limits, but I don't think "proved mathematically" is a good way to state that. You can't "mathematically prove" a theory agrees with reality except by reconciling it with experiments (possibly indirectly through links to previous theories), so t…

Did you google it? https://en.wikipedia.org/wiki/Mathematical_proof You're missing the point of what I said w/ the Einstein example. He proved it via maths. Whether or not it was empirically validated is irrelevant. It was true mathematically Today, this cannot be done with fields like medicine and psychology. No one can create a mathematical proof for a cure for cancer, make a treatment based upon that proof, and ha…

I've done enough mathematical proofs that I don't think I need to "google it", but I read the Wikipedia page anyway. It failed to explain how a mathematical proof could have anything to do with "proving" General Relativity in absence of empirical evidence. He certainly proved that (as I said before) it agreed with existing (experimentally backed) theories using mathematics, but if you have a method which allows you to deduce the truth of physical laws from purely mathematical proofs (no links to experiments at all!) you'll revolutionize epistemology. As it stands most scientists are using the crutch of experimental evidence at some level (either directly or by reasoning within theories that are at least for now validated by experiment).

> Whether or not it was empirically validated is irrelevant. It was true mathematically

If you mean it was a self-consistent theory, note that there are infinite physically incorrect theories which are "true mathematically". It is true mathematically in the same way number theory is "true mathematically", but you don't see anyone assume that we can describe gravity with prime number theory. If you mean it was mathematically proved consistent with previous physical theories backed by evidence, we're back to the experimental link, and also there are infinite physically incorrect theories which would agree with Newtonian mechanics and special relativity. If you mean he proved it satisfied some properties we expect from physical theories because they've been consistently upheld (e.g. energy conservation), there's an infinite number of wrong ones there too.

Mathematical convenience has been an excellent guide in physics, particularly fundamental physics (electromagnetic waves, antiparticles, and the Higgs Boson were all the results of positing things for mathematical convenience), but it is not a substitute for verifying novel experimental predictions. Nobody taken seriously in the physics (or mathematics for that matter!) community thinks it is, not even the often decried string theorists.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#226

Post by kazinator »

throwaway729 wrote: post #103
return0 wrote: post #79

Earlier quoted context omitted.

We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences…

> we need to foster a culture that a) accepts repetition Do we? I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use…

Results can be irreproducible even if nobody was dishonest.

Reproduction is a way of bolstering rigor.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#227

Post by jsprogrammer »

misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

>Repetition is good, of course, but isn't a replacement for good science in the first place.

Repetition is science.

>Define repetition.

Google says: "the recurrence of an action or event."

Science requires 1) a prediction of an observation 2) repeated observation consistent with the prediction.

If there is no repeated observation of what was predicted, it is not science.

>But everyone believes the results, and believes that they represent the Higgs.

It doesn't matter what is believed. "Everyone" used to believe all kinds of things, that doesn't mean that those things were true, or accurate. Repeated observation of prior predictions is all that matters.

>When it costs billions to construct your experiment, sometimes reproducing the exact same thing can be hard.

That doesn't excuse, or allow in, things that haven't been reproduced.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#228

Post by adrianm »

throwaway729 wrote: post #139
devishard wrote: post #138

Earlier quoted context omitted.

> I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. No one is proposing repetition for its own sake. The point of repetition is to create rigor, and you can't do rigorous science without repetition . > Paying for the cost of mountains upon mountains of lab techs and materials that it w…

I could see a valid argument for only doing science that will be worth replicating, because if you don't bother to replicate you aren't really proving anything. Exactly. A lot of the science I've done should not be replicated. If someone told me they wanted to replicate it, I would urge them not to. Not because I have something to hide. But because some other lab did something strictly superior that should be replica…

This is a really good post which contributes to the conversation. Why make it on a throwaway account? We need more of this here!

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#229

Post by Armisael16 »

andrepd wrote: post #213
noobermin wrote: post #206

Earlier quoted context omitted.

There is a philosophical assumption underpining astronomy and cosmology, the so-called "cosmological principle."[0] This is an assumption (albeit not completely unfalsifiable) that physics across the universe at large scales is the same for all observers, that here on earth there should be similar to physics in andromeda. Of course , all scientific experiments done have been done on or near earth, and thus, our inter…

But we can measure gravity in faraway galaxies. We don't need to actually go there, we have telescopes :)

How can you measure the strength of the gravitational constant if you don't know the masses of the objects in question?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#230

Post by anotherhacker »

johncolanduoni wrote: post #225
anotherhacker wrote: post #216

Earlier quoted context omitted.

Did you google it? https://en.wikipedia.org/wiki/Mathematical_proof You're missing the point of what I said w/ the Einstein example. He proved it via maths. Whether or not it was empirically validated is irrelevant. It was true mathematically Today, this cannot be done with fields like medicine and psychology. No one can create a mathematical proof for a cure for cancer, make a treatment based upon that proof, and ha…

I've done enough mathematical proofs that I don't think I need to "google it", but I read the Wikipedia page anyway. It failed to explain how a mathematical proof could have anything to do with "proving" General Relativity in absence of empirical evidence. He certainly proved that (as I said before) it agreed with existing (experimentally backed) theories using mathematics, but if you have a method which allows you t…

Interesting points. I still disagree, or perhaps I'm not doing a good job explaining.

My point is this: Something is true whether one accepts it or not. E.g. the earth was proven to be round before anyone actually went around it.

Another example is climate change deniers. To them there's no proof of; 1) Climate change; 2) That it's man made.

Another example is people who believe the earth is 10,000 years old (I'm not joking millions of ppl believe this). They will deny any evidence you put in front of them.

That's the beauty of maths. If you can prove it mathematically, it's true. Whether or not you believe it is irrelevant.

Post reply on HN