Keeping computers from ending science's reproducibility

[is that frossie frossie? hi!]

i'm not sure what problem you're talking about - someone reproduced the results with separate code and data, so what's to worry about? (i don't mean that because it was confirmed it was ok, but rather that if it had been wrong, we would have known in the end... after all, people make mistakes all the time - science is a collective enterprise that relies on many overlapping, interlocking pieces)

"Weather simulations have even more data; they are less accurate for other reasons."

I doubt anyone can beat particle physics for sheer information quantity. I seriously doubt that we have petabytes of real weather information to feed our simulations. I can find some references online to petabyte stores for weather simulation results online, but that's ultimately just cache, not information (in the information theoretic sense).

This is part of what I mean when I talk about the information theory, and how you can't get more information in the information-theoretic sense than the sum total of the simulation and the original data. Weather simulations may chew through terabytes or petabytes of RAM in the simulation phase, but they are not fed that much data. If the people involved mistake it for real information, then this is also part of what I mean when I say that once you get into using computers in a big way my training does indeed start giving me standing to complain again by even the most rigid "stay out of my science" standards.

Secondly, your assumption that running the simulation again is useless in a world where you can casually assume that you have all the data and the simulation and can have the contempt bred of familiarity for the whole process. In the real world, if you can't replicate the results, how can you criticize the model? I think there's still some "the computer said it, it must be right" underlying your answer; you can't assume the computer model is worth anything, it must be proved and debated and peer reviewed, which is not possible if you can't even get it to run and get the same results. The model is still subject to scientific inquiry, it can't be given a free pass.

Independent replication of results is also desirable, but you need both. In a world where nobody can replicate the results and it's hard to verify the simulation against the real world, it's too damned easy to end up with the Feynman electron mass situation where the selection effects from the researchers dominate the putative results of the simulations. The researcher summaries of the results of some runs of some models you can't see or execute and some data you can't get at happen to align... what does it mean? Frankly, who knows?

I'm married to a biology PhD, and at one point spent a couple of years watching people, including biologists, deal with a constant stream of Creationists in places like talk.origins.

My experience is that if you're a personal friend, you get more serious conversation. If you're someone they know but not so closely, they'll have the argument if pushed but don't feel the need to actively educate. And if you're a random person spouting on the Internet, it isn't worth their time to get involved. If pushed to be involved, they don't feel the need to be pleasant about it.

After spending time myself explaining the same thing over and over again, I've come to feel the same way. I've also come to realize that there are plenty of smart people who do not wish to be educated. Including in my direct personal experience, at least one PhD in mathematics and another in molecular biology.

Based on this experience I have some sympathy for the position of climate scientists. You spend your life climbing around glaciers in Greenland, and you don't really feel like spending the rest of it convincing people who don't want to bother learning the basics about climate.

Several satellites take high resolution real time pictures of global weather they run 24 x 7 for years and that's RAW data. There are several of these plus radar stations etc. NOAA uses a subset of that information to make weather forecast (they normally toss out old data and data from the other side of the planet because it's not useful even if it might make a slightly better forecast). We can make highly accurate block by block forecast an hour ahead over major cities and just about any point in the US. However, there is little point in reading that level of detail from a forecast for such a short period of time. Sometimes when the forecast is 50% chance of rain over the next 8 - 12 hours they know where and when it's going to rain they just don't know where you are.

edit: One of the world's largest scientific data systems, NASA's Earth observing system data and information system (EOSDIS) has stored over three petabytes of earth science data in a geographically distributed mass storage system. that's just Nasa and a lot of their data does not make it into EOSDIS.

The goal of science is to understand the world. Running the same simulation on the same data and getting the same result only tells you that the machine running the simulation is not broken. What you want is to run a different program with different assumptions on different data and come to the same conclusion. This is actually used to make 7 day forecasts. They run a few different models with different assumptions and pick the average result. Over time each model is updated independently to maintain its independence.

PS: You don’t validate E=MC^2 by doing the exact same experiment 10,000 times. You do every type of experiment that you can think of which relates to E=MC^2 looking for anything which does not work out the way you think it should.

I think this ongoing discussion is fascinating, and I don't disagree with you.

But (and you knew that was coming), re-running someone else's code, or reproducing someone else's experiment is verification. It doesn't mean that what they did is valid, but it verifies that they did do what they said they did.