Keeping computers from ending science's reproducibility

It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researchers have any clue what door they are walking through? How many of them know what a strange attractor is? I'm sure the answer is non-zero; I'm equally sure the answer is nowhere near 100%.

Small errors cascade even if you consider a non-chaotic classical model. (That is, not that there is such a thing as an iterative model that is not potentially subject to chaos, but rather than even if you don't understand chaos you can see that small errors can cascade. Chaos just makes it worse, and weirder.) A simulation will have bugs like any other large problem. A non-programmer approaches bugs by banging on the program until it seems to generate expected results. (About 50-80% of programmers do that too.) Therefore, many of these simulations are simply reflections of the simulator's expected result, due to the effect of the researcher's selection mechanism running on the results of the simulations they run. How do we verify that this is not the primary factor in the result of the simulation? This need not be conscious. It need not be ideological, either; I can easily envision a simulation that "should" return a boring or trivial result being monkeyed with until it produces something "interesting", because the simulators think the boring result should not obtain.

A lot of algorithms you can use in these simulations are fundamentally unstable when used iteratively; some exacerbated by floating point errors, some mathematically unstable even with perfect real numbers. How many of these simulations use something unstable without even realizing it, given that it could take a professional mathematician to work out whether that's the case? Even algorithms thought to be stable and reliable can fall apart under pathological situations, and one of the odd things about mathematics is just how often you end up hitting those pathological situations when programming; far more often than it seems like should be the case.

In information theory terms, a simulation can not contain more information that the sum total of the input data and the content of the simulation algorithm. How many simulators understand the full implications of that statement? I sure don't understand the full implications of that, but what I do understand makes me pause a bit. Very simple simulations with rules that can be verified and initial data that is very solid I can deal with; for instance, I like the cell-automata based social theories that show the spread of information or political views or something, especially when it is clear the researchers understand that it's only an approximation. But as the initial data starts getting sketchy or the simulation grows enormous, I start getting nervous about the actual information content of the output. Just because the output appears to be information doesn't prove that it is. It is vitally necessary to be able to check the simulation against real data. For instance, physical simulations of, say, cars crashing can be verified. How many simulations can actually be verified, though? Frequently the reason computers were reached for in the first place is the inability to do the real experiment. Any simulation that can't be verified should be presumed worthless by default. How often does that happen? (It's 20-f'ing-10 and "the computer said it, it must be right" still runs rampant through our culture....)

And of course there's the whole reproducibility issue, where the absolute bare minimum for science would be to publish the full simulation program, all data, the necessary invocation and compile instructions to bring the two together, and all necessary information to understand the input and the output. Clearly, this is not something that fits in a journal paper, but how often does this happen at all?

No, I am not referring to any specific discipline here and in particular I'm not actually referring to climate science. I'm nervous about the whole movement towards simulations in general.

Note that I'm not reflexively against the idea. Meet these bars and I'm happy; give me enough data for reproducibility and verify that your simulation is in fact simulating something real and corresponds to reality and I am happy. (Many physical simulations fit in here.) But as more disciplines jump in I am concerned that these bars are not well understood, and I'm seeing ever more press releases about simulations that can't possibly meet these bars.

It wasn't always this way, unfortunately. I've been involved with trying to glean some formulaic/methodological insights from spreadsheets and code and it's not always possible to reverse engineer the essential methodology or be sure that mistakes were avoided.

Proper scientific research practices are similar to proper data backup practices: the documentation (backup files) are important, but they don't matter if you can't have successful reproducibility (restoration).

If you can get buy-in from the research community, even in one niche field, I could see great utility in a central escrow house of sorts, where people can post research software, datasets, etc. and others can download the stuff and validate the results only after agreeing in some reasonably binding way not to use that information to scoop or copy the original researcher.

Buy-in in this case means that everybody should have to do it to publish in a certain venue. You might even do something fancy like providing people access to a VPS which will run the server without allowing validators to scp everything back to their own machines. Of course this is impossible to do perfectly; once you allow people to view the data in a text editor or hex dump, it is all over in a strict sense (scripts + screen-scraping), but it is still a step up from just handing over the files, and keeps honest people honest.

You might also provide some anonymization/randomization service for datasets (for instance, by applying an unknown linear transform to absolute numbers in the case that the results don't change, similar to what was done here: http://googleresearch.blogspot.com/2010/01/google-cluster-da... .

I think there are 2 reasons why researchers can be reluctant to share:

1) They are afraid of getting scooped by somebody else leveraging all the effort they put into developing the tools and curating the data and leapfrogging them.

2) Their results are suspect.

Such an escrow house would ameliorate the concern behind 1) and help suss out 2) :)

Anyone else have thoughts on this?

I talked myself blue in the face trying to explain how science doesn't work if you don't give people enough information to reproduce your research! I couldn't get him to understand though. Arggh so frustrating.

i had a bunch of perl scripts (ah, those were the days) that mangled various files before feeding them into fortran least-squares stats code that took a day or so to run.

by the end, it was pretty much chaos. i was a self-taught programmer, these were probably the second or third "significant" programs i had ever written. nothing was documented, everything took so long that i couldn't check much... i had bugs, of course.

in the end i published. maybe 6 months later i got an email from someone in the states. they were trying to reproduce my results. in the end, they did (as far as i know).

so the system worked.

incidentally (perhaps the only useful point here) they must have used different data. that's something worth explaining in more detail - "my" data came from years of painstaking work by a bunch of people working for my thesis supervisor. yet 6 months later the results could be duplicated from a week or so of data from a much more powerful telescope (the keck). so data in research often aren't as critical as you might think. things progress at such a rate that even if you don't share data, it's trivial to reproduce just a short time later... (and i am pretty sure that this is true in gene sequencing, for example)