Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

Post reply on HN
Print view View original

61–70 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#61

Post by eru »

dalke wrote: post #54
guaka wrote: post #2

Totally agree. I'd go even further and make free licenses on scientific source and datasets mandatory. Research that is funded by public money should lead to public code and data.

So, $1 of public funding triggers public code release? Or is there a threshold? No one has been able to tell me why the need for reproducibility requires software freedom. Consider the program 'nauty'. It is available in source code form for anyone to review, but it cannot be used for military purposes. That's not free, certainly. But isn't that enough to call it good science? Similarly, consider the clause "only for…

> So, $1 of public funding triggers public code release? Or is there a threshold?

Yes, 1$, no threshold necessary.

For projects where the cost of publication of data is not worth the dollar, people will just not accept the dollar. (That's where a 'natural threshold' comes in.)

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#63

Post by dalke »

ebbv wrote: post #44
dalke wrote: post #35

I disagree. One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. (Even if something like it is detected in the future, is it really a "repeat"?) Some experiments are immoral to repeat. For example, in a drug trial y…

> One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. First off, you can accept the observation at face value as an observation, but conclusions drawn from the claims which have no other support or means of verific…

> conclusions drawn from the claims which have no other support or means of verification should not be accepted and would not be accepted

I disagree. In all cases, even with repeated experiments, the claims are only tentatively accepted. The confirmation by others of Blondlot's N-rays didn't mean they were real, only that stronger evidence would be needed to disprove the conclusions of the earlier observations.

Astronomy papers make conclusions based on rare or even singular observations. Take SN1987a as an example, where observations from a neutrino detector were used to put an upper limit on the neutrino mass, and establish other results.

> "or performing another test"

This question is all about repeating an experiment. Repeating the experiment would be immoral.

There are certainly other tests which can confirm the effectiveness, without repeating the original experiment and without being immoral. For the signal strength I gave, we can compare the treated population to the untreated population using epidemiological studies.

But under current medical practices, if a drug trial saw this sort of effectiveness, the trial would be stopped and everyone in the trial offered the treatment. To do otherwise is immoral. As would repeating the same trial.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#64

Post by eru »

hudathun wrote: post #38
wccrawford wrote: post #23

Earlier quoted context omitted.

We have some kind of weird hero-worship of scientists where the general public just believes what they say, even if they never even attempt to replicate their results. They do an experiment (which may or may not be scientifically sound to start with) and then publish results, and the public eats it up. And then people have the nerve to say, "Last week chocolate was bad for me, now it's good? Make up you mind!" No, st…

Good point about the public involvement. The public and the news systems are part of the problem. I've lost count of how many 'battery breakthrough' articles I've come across, but they seem to pass the newsworthy test.

Wasn't the problem with battery breakthroughs that they don't commercialise well, rather than that the science doesn't repeat?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#65

Post by eru »

grashalm wrote: post #14

Sometimes in cs if your research is embedded in a huge ecosystem, it can become quite expensive to reproduce results. I mean proper reproduction, not just rerunning the Benchmarks. If you are dealing with complicated stuff, the reproducer might also just not be able to do the same thing technically.

Just being able to rerun the benchmarks (or other data analysis for non-CS papers) would be an improvement on the current state of affairs, where people often don't publish code nor data.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#66

Post by misnome »

ebbv wrote: post #44
dalke wrote: post #35

I disagree. One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. (Even if something like it is detected in the future, is it really a "repeat"?) Some experiments are immoral to repeat. For example, in a drug trial y…

> One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. First off, you can accept the observation at face value as an observation, but conclusions drawn from the claims which have no other support or means of verific…

> But even something that is rare and relies on observation, like gravitational waves, we have already been able to identify more than one occurrence.

But who didn't believe the original result? And does the same experiment observing multiple occurences really count as 'reproducability'?

We've only had ONE observed local group supernova within the past several hundred years (and that was within our lifetime, thankfully with several relevant detectors up and running). Should we ignore any result or conclusions from this instance?

No - if the data from the next supernova disagrees or reshapes the field - and it probably will, given the huge amounts of resources dedicated to studying it (see e.g. http://snews.bnl.gov/), this will just be evidence of scientific progress - reshaping your position based on experimental data.

Again, I think there is a certain amount of crosstalk with people who say that the "Entire community of scientists" has a problem, whilst actually meaning specific fields. Perhaps an ironic imprecision.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#67

Post by tnhh »

dagw wrote: post #43
wodenokoto wrote: post #33

Earlier quoted context omitted.

I recently used a large dataset of tweets in a research project. As far as I know, I do not have the rights to distribute these. I also used a dataset consisting of newspaper articles. It cost me $1.000 to get access to, and I definitely do not have the rights to redistribute it.

As long as you provide a detailed enough description of the source of your dataset that I can reproduce it myself then that is fine. So in your first case tell me what criteria you used to select your tweets and in the second tell me where to send my $1000 and what to ask for.

Unfortunately not everyone reports this information. Here is a study that we did of over 500 papers using online social network data: http://tnhh.org/research/pubs/tetc2015.pdf While most authors would report high-level characteristics (e.g., which social network they measured), fewer authors reported how they sampled the network or collected data, and very few people reported on how they handled ethics, privacy and so forth.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#68

Post by misnome »

dalke wrote: post #63
ebbv wrote: post #44

Earlier quoted context omitted.

> One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. First off, you can accept the observation at face value as an observation, but conclusions drawn from the claims which have no other support or means of verific…

> conclusions drawn from the claims which have no other support or means of verification should not be accepted and would not be accepted I disagree. In all cases , even with repeated experiments, the claims are only tentatively accepted. The confirmation by others of Blondlot's N-rays didn't mean they were real, only that stronger evidence would be needed to disprove the conclusions of the earlier observations. Astr…

I love that both of our replies used 1987A as an example...

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#69

Post by tnhh »

nxzero wrote: post #13
middleman90 wrote: post #5

I thought this was in the definition of "scientific"

Peer review is how most science is defined as science and peer review does not require reproduction of the work.

Peer review is a relatively new aspect of science: the Philosophical Transactions of the Royal Society started systematic peer review in the mid-19th century https://arts.st-andrews.ac.uk/philosophicaltransactions/brie...

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#70

Post by tnhh »

wodenokoto wrote: post #36
bontoJR wrote: post #11

The big issue right now is funding of replicated research, who wants to fund a research to prove someone else was right? Most of these funds are granted based on potential outcome of the new discovery like: potential business, patents, licenses, etc... not being the first one would probably wipe most of these benefits, cutting down to a small probably getting funded... Now, straight to the point, who's going to pay f…

On a low level, I think it should be mandatory for masters students to do a pre-thesis project, which is replicating findings in a published paper. It would do something about low hanging fruit in terms of testing reproduceability and since there is a published paper, the student has access to guidelines for setting up and reporting on a large project, which will help them learn how to do their own, original thesis.

I had my Masters students do this as part of my wireless networking class this year. It was very instructive for me and the students seemed to enjoy it, so I'll definitely keep it in the syllabus.
Post reply on HN