We Should Not Accept Scientific Results That Have Not Been Repeated • Hacker News Board

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#251

Post by toufka » Sat, Jul 30, 2016, 1:40 AM UTC

Obi_Juan_Kenobi wrote: post #192

zouhair wrote: post #81

Earlier quoted context omitted.

You ask that like it is something new. Reproducibility[0] is something at the core of science. The problem these days is that spending time doing replication is not glamourous and will not help you get funds. [0]: https://en.wikipedia.org/wiki/Reproducibility

The problem is worst in fields that don't have reproducibility 'built in' to the field. I do genetics and development, and the main sanity check we have is the distribution of mutant lines. If you say that mutant X does Y, other people are likely going to see that (or not) when they get their hands on it and start poking around. This strength of working with mutants is at the core of the success of molecular biology.…

Exactly. Most 'good science' start with premises A, B, and C from their respective papers, and go on to show that X is likely true because of this new data. In general, it would be impossible to propose X if A, B, and C were not themselves true - they were the foundations of this new research. In that way, even though the publication of X may not have directly (though often precisely directly) repeat the experiments of A, B, and C, they still have replicated the core concepts.

As noted, this does not get rid of the confounding negative effects, where paper D was also used as a (in retrospect, faulty) premise as well. Though no one ever actually comes out and says 'D' is bad.

Over time, A, B, and C theories accrue significant weight while D falls off. It's not as explicit as some may like, but in the end the spirit of replication is well and alive.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#252

Post by ipstone2014 » Sat, Jul 30, 2016, 1:53 AM UTC

return0 wrote: post #3

We now have the tools to do it , and we should be doing it. The fate of scientific findings is not to publish papers, they belong to open and continuous scrutiny. And someone should build a github of scientific facts.

let's try it, if interested, send me a tweet at @isaacpei, seriously thinking about creating something for this

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#253

Post by lutorm » Sat, Jul 30, 2016, 3:38 AM UTC

Define reproduced? Do we mean "conduct the same experiment multiple times so we can assess the variance on the outcome"? Or do we mean "conduct the same experiment multiple times to figure out if the first result is a screw-up"?

Those two aren't the same, and I think far too many think that the point is the latter when, imho, it's actually the former. Pure screwups will likely get found out, just like glaring bugs are usually found. It's when your result actually has a huge variance but you're looking at only one (or a few) samples and draw conclusions from it that's insidious, like the fact that it's the bugs that just change the output by a tiny bit that are the hardest to notice.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#254

Post by ycombinatorMan » Sat, Jul 30, 2016, 6:03 AM UTC

sn9 wrote: post #248

euyyn wrote: post #246

Earlier quoted context omitted.

The very title claims that unrepeated results shouldn't be accepted .

The contents of the article are the only thing that ever matters. The title is picked by editors to get the most people to click on it.

I think the title is all that matters. I'm only here for the discussion, and the title defines the topic.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#255

Post by eru » Sat, Jul 30, 2016, 7:47 AM UTC

dalke wrote: post #72

eru wrote: post #61

Earlier quoted context omitted.

> So, $1 of public funding triggers public code release? Or is there a threshold? Yes, 1$, no threshold necessary. For projects where the cost of publication of data is not worth the dollar, people will just not accept the dollar. (That's where a 'natural threshold' comes in.)

So, if I have a summer intern who is paid by the government, and helps on my otherwise non-government-funded project, then is that enough to trigger the requirement? Or, if I use a government facility, like a supercomputing center, then that also trigger the release requirement? Even if no money changes hands? What about a government network? It sounds very tricky. If I get a government grant to work on a project, an…

Just set up some very clear and simple rules. Don't worry about small threshold when setting up the rules, simplicity trumps.

(Because corner cases where small amounts of funding would trigger the requirements can be worked around by just not using that find. As long as the circumstances triggering the requirements are easy to predict.)

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#256

Post by austinjp » Sat, Jul 30, 2016, 9:11 AM UTC

Mendenhall wrote: post #136

throwaway729 wrote: post #133

Earlier quoted context omitted.

I suppose I should remind you, then, that the up-thread comment we're responding to was written in the context of a physics experiment. And I'm about as far from "soft sciences" as you can get.

Oh I was not commenting on what you do. I was commenting that many who hold that view of non repetition have leanings that are favored by soft sciences. They dont feel the "need" for repetition even in hard science. It is a personal bias thing that influences their need. It can happen when emotion overcomes hard logic and science.

> many who hold that view of non repetition have leanings that are favored by soft sciences. They dont feel the "need" for repetition even in hard science.

Really? You're suggesting that psychologists (to arbitrarily pick a softer science) would deny physicists (to arbitrarily pick a harder science) should reproduce their studies where possible?

That seems remarkable to me, perhaps I've missed these discussions. Can you provide evidence that this is a pervasive movement in some sciences, rather than the opinion of a few?

"Many" is a trigger weasel word, of course, and needs backing up.

My interpretation -- perhaps incorrect -- is that you feel the softer sciences are wilfully undermining the quality of harder sciences. I very much doubt this is the case. Some philosophers of science and some softer science key influencers may introduce difficult and challenging questions about the appropriateness and usefulness of some research methodologies (as are people in this thread) but I doubt they'd make the blanket assertion you're suggesting.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#257

Post by jernfrost » Sat, Jul 30, 2016, 9:58 AM UTC

odbol_ wrote: post #186

mmierz wrote: post #120

I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research p…

This is why I'm always skeptical of people zealously claiming "all GMOs are perfectly safe! It's been proven !" Yeah, proven by expensive studies that were funded by the company making the GMO. Who is going to pay for another study to try to disprove it? There's a reason we sprayed DET all over our vegetables for years before it was banned: there was no scientific studies proving that it was harmful, even though it c…

Sorry but this is such a fundamentally wrong assertion on so many levels. Quite the contrary people ought to prove that there is a good reason to assume GMOs are inherently more unsafe than anything else. It is the anti-GMO crowd who are the creationists here relying on emotions and dogma.

GMO is controlled alteration of specific genes in a crop. Why on earth should we assume this is less safe than uncontrolled changes? Massive changes of crop DNA has been going on for decades. Food you eat every day has originally been produced by essentially nuking seeds and making lots of random mutations. One has then grown various "nuked" seeds looking for those with favorable characteristics. Plants get changed getting genes from bacteria, cross breeding and all sorts of things. Why on earth should be assume uncontrolled massive changes of plant DNA is somehow safer than GMO?

GMO is not really what needs to be proven. It is the other methods of changing plants which need to prove that they are safer than GMO.

The problem with GMO isn't the science of it. The problem with GMO is the politics. Here I agree with the anti-GMO crowd. Companies like Monsanto are highly immoral and a scourge of the earth. It is companies owning particular strains of seeds and requiring everybody to keep buying from them which naturally causes resentment.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#258

Post by jernfrost » Sat, Jul 30, 2016, 10:08 AM UTC

I don't get why this is not top on the agenda for the scientific community and the government. Huge amounts of research money is lost in repeating stuff that doesn't work. Huge amounts of money is lost chasing broken science.

I blame this on the neo-liberal ideology. This intense focus on getting money's worth, on tying grants to specific goals, counting publications etc. Driving research exclusively on a very narrowly defined money incentive has driven us further into this sort of mess. The money grabbing journals which has prevented any significant innovation in how science is shared.

I think what science needs is a model closer to that of open source. With open projects anybody can contribute to but where verification happens through personal forged relationships. The Linux kernel code quality is verified by a hierarchy of people trusting each other and knowing something about each others quality of work. Work should be shared like Linux source code in a transparent fashion and not behind some antiquated paywall.

I don't think the grant system can entirely away, but perhaps it should be deemphasized and instead pay a higher minimum amount of money to scientists for doing what they want. Fundamental science breakthrough doesn't happen because people had a clear money incentive. Neither Einstein, Nils Bohr, Isaac Newton or Darwin pursued their scientific breakthroughs with an aim of getting rich. Few people become scientists to get rich. Why not try to tap into people's natural desire to discover?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#259

Post by return0 » Sat, Jul 30, 2016, 10:45 AM UTC

ipstone2014 wrote: post #252

return0 wrote: post #3

We now have the tools to do it , and we should be doing it. The fate of scientific findings is not to publish papers, they belong to open and continuous scrutiny. And someone should build a github of scientific facts.

let's try it, if interested, send me a tweet at @isaacpei, seriously thinking about creating something for this

There is no lack of efforts to get scientists discussing (shameless related plug http://sciboards.com), but unfortunately there is a disincentive for scientists to do so (politics).

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#260

Post by LizardLarry » Sat, Jul 30, 2016, 10:50 AM UTC

tstactplsignore wrote: post #212

odbol_ wrote: post #186

Earlier quoted context omitted.

This is why I'm always skeptical of people zealously claiming "all GMOs are perfectly safe! It's been proven !" Yeah, proven by expensive studies that were funded by the company making the GMO. Who is going to pay for another study to try to disprove it? There's a reason we sprayed DET all over our vegetables for years before it was banned: there was no scientific studies proving that it was harmful, even though it c…

This is just a large strawman argument. Talking about the safety of GMOs is more like talking about the safety of "Medicine"- you're actually talking about the safety of a large variety of different products. Pushing the narrative that all GMOs should be treated equally is like pushing the narrative that all medicines should be treated equally, which is nonsensical. The very idea that there can be "something wrong wi…

>The very idea that there can be "something wrong with GMOs" is as anti-scientific

Are you kidding me? Have you read Taleb on the risks of GMOs? He argues that the possibility space of danger from GMOs is different from the possibility space of danger from regular breeding.