Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping computers from ending science's reproducibility

arstechnica.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Keeping computers from ending science's reproducibility

#51

Post by snth »

jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

Can you provide references that would allow a researcher to educate himself? I'm genuinely interested, and there seem to be lots of fields that use software in experimentation and simulation but where there isn't much emphasis or education on doing it properly.

Re: Keeping computers from ending science's reproducibility

#52

Post by cynlillbox »

jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

jeff:

i'd like strongly to encourage you to offer your opinions on things on which you are nonexpert. having only experts allowed to opine, with experts also relaining monopoly on experts, leads to echo chambers.

however, you seem to be reacting to a different article entirely. nowhere did the article mention simulations. if you look at Stodden's CV (the woman whose talk motivated the article) she's a statistician. The need for code-sharing is in fields like climate and bioinformatics where people's analyses lead to changes in politics, policy, pharma/health, etc; not for people deciding whether lorenz goes chaotic at precisely the right bifurcation parameter value.

more and more and more science is becoming statistical and data-driven. chaotic dynamics, though very important, is not the issue here. it's analysis of complex data coming from real-world systems.

there are many interesting and wise comments in this thread but they are almost all about an article on simulations -- ie. not the article the OP points to.

Re: Keeping computers from ending science's reproducibility

#53

Post by jimbokun »

Retric wrote: post #36
yummyfajitas wrote: post #32

Earlier quoted context omitted.

On the contrary, I think math ed types want to keep the actual mathematicians out precisely because the mathematicians recognize how complex the field is. Every time I've been told "stay out, you don't know our field", it was because I pointed out complexity or plausible alternative explanations. For example, a puzzle: why do SAT scores underpredict female performance in Calc 1, and overpredict grades in higher level…

Honestly, SAT scores vs. Performance is the type of (mostly bad) research I am talking about. The SAT is not designed to test math ability. It's focus is on how likely a student is to finish their freshman year of college and it does that fairly well. You can do a lot of useless research in this area and it tells you next to nothing. If you want to predict a highschool students ability in advanced collage math classe…

"There are significant and useful study’s which suggest 2 hours of nonstop instruction is less useful than two one hour periods with a moderate break in-between."

The professor of the last class I took was very conscientious about taking a break at the 1 hour mark of an 80 minute lecture.

Re: Keeping computers from ending science's reproducibility

#54

Post by wmf »

sparky wrote: post #9
jvdh wrote: post #3

A similar argument actually goes for experiments that are somehow affected by computer networks. If scientists use grid-computing, cloud-computing, or just the plain regular Internet, there is no way to accurately reproduce results for distributed applications. Luckily, some researchers are aware of this and there are now some projects starting to make testbeds and infrastructures to make environments where experimen…

Links, for the curious: http://www.planet-lab.org http://www.hpl.hp.com/open_innovation/cloud_collaboration/cl... http://cloud.cs.illinois.edu/ http://www.cs.duke.edu/courses/spring08/cps214/project.html (Testbeds and Emulation section)

PlanetLab and the clouds are great ways to do non-reproducible research, because the other users in the system (where sometimes "the system" is the public Internet) create interference. Reproducible networking research either has to be simulated or has to be run on an isolated testbed such as Emulab.

Re: Keeping computers from ending science's reproducibility

#55

Post by earthboundkid »

jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

“It is vitally necessary to be able to check the simulation against real data.”

Actually, that’s not enough either. In spam training, you have to have a completely set of data that you use for the final test but never, ever for training. Why? Because otherwise your spam filter will learn how to properly classify every message you train it with and nothing else.

For simulated science, this means it’s not safe to say, “Well, we trained the simulation until it could reproduce 2000-2010 when we gave it 1990-2000 as input data.” If you do that, your predictions for 2010+ are probably going to be worthless, since your simulation is like a cheating student who only knows the right answers on old tests he stole, not how to pass any test in general.

Re: Keeping computers from ending science's reproducibility

#56

Post by sparky »

wmf wrote: post #54
sparky wrote: post #9

Earlier quoted context omitted.

Links, for the curious: http://www.planet-lab.org http://www.hpl.hp.com/open_innovation/cloud_collaboration/cl... http://cloud.cs.illinois.edu/ http://www.cs.duke.edu/courses/spring08/cps214/project.html (Testbeds and Emulation section)

PlanetLab and the clouds are great ways to do non -reproducible research, because the other users in the system (where sometimes "the system" is the public Internet) create interference. Reproducible networking research either has to be simulated or has to be run on an isolated testbed such as Emulab.

The results from PlanetLab et al. may not be deterministic, but you should be able to reach the same conclusions based on repeated experiments. Otherwise, your results may be a little too fragile to form the basis of sweeping conclusions.

This may not be ideal, but it is no worse than any branch of science which is not purely digital.

Re: Keeping computers from ending science's reproducibility

#57

Post by joe_the_user »

lutorm wrote: post #17
jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

I certainly agree with you that there is a lot of publication of results generated with computers that is non-reproducible. However, I'm not sure publishing the complete codes that were used is necessary, as long as there is a detailed description of the algorithm used. In essence, I think it's essential that multiple people write their own codes and reproduce the result. The danger in publishing your code is that it…

The danger in publishing your code is that it is easier for someone to "reproduce" your study by just taking my code and running it, and there is a danger in having the community rely on a small number of codes.

Having a small number of codes seems like an excellent situation. What you should have is large of tests and much test data to test the small number of codes.

Testing code in thorough manner is an important addition to looking at it.

Re: Keeping computers from ending science's reproducibility

#58

Post by frossie »

andrewcooke wrote: post #41
frossie wrote: post #30

Earlier quoted context omitted.

But this is a problem that can be tackled, and those who take it seriouslt already do so. For example, in our pipelines we use an infrastructure that always adds every command executed on the file, with every exact parameter, to the metadata of the file, starting from one canonical archived file - and hence, one can indeed reproduce manually the result of the pipeline given sufficient time and dedication. [Edit: we a…

[is that frossie frossie? hi!] i'm not sure what problem you're talking about - someone reproduced the results with separate code and data, so what's to worry about? (i don't mean that because it was confirmed it was ok, but rather that if it had been wrong, we would have known in the end... after all, people make mistakes all the time - science is a collective enterprise that relies on many overlapping, interlocking…

[Yeah, didn't I share an office with you 15 years ago? :-) ]

But data can't always be reproduced - Shoemaker-Levy won't hit Jupiter again any time soon, and while big results will cause a rush for verification, the little steps are usually believed as is. Moreover astronomy is a special case where - let's face it - if we go down the wrong path for a decade or so, nobody's bleeding.

Take on the other hand the processing of climate data - if crap engineering (not malice) causes garbage to come out of the data, this is a problem for everybody. So I think the OP is right in that proper auditing of computer-processed science data should be possible, I was just pointing out that it's not as hard (technically) as people think to achieve.

Re: Keeping computers from ending science's reproducibility

#59

Post by jwhite »

sparky wrote: post #9
jvdh wrote: post #3

A similar argument actually goes for experiments that are somehow affected by computer networks. If scientists use grid-computing, cloud-computing, or just the plain regular Internet, there is no way to accurately reproduce results for distributed applications. Luckily, some researchers are aware of this and there are now some projects starting to make testbeds and infrastructures to make environments where experimen…

Links, for the curious: http://www.planet-lab.org http://www.hpl.hp.com/open_innovation/cloud_collaboration/cl... http://cloud.cs.illinois.edu/ http://www.cs.duke.edu/courses/spring08/cps214/project.html (Testbeds and Emulation section)

NICTA and WinLab have developed a management framework called OMF for specifying and running experiments on testbed networks. One of the main goals of the project is to improve reproducibility in networking research.

http://omf.mytestbed.net

(It is mainly written in Ruby)

Post reply on HN