Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping computers from ending science's reproducibility

arstechnica.com

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Keeping computers from ending science's reproducibility

#21

Post by kasterma »

yummyfajitas wrote: post #5
jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

I am often told that I should keep my nose out of other science domain's business because they know more than I do. I've only heard such statements coming out of a few fields: math education, labor economics, climate science and psychometrics of race/gender. You should ignore such statements; they are nothing more than an attempt to bully you into accepting received wisdom from activists with a PhD. As an actual scie…

As another scientist I have to strongly second this. Some things might take real work on your part to understand though. Even within different fields of mathematics there are difficulties in understanding (complaint about this: http://www.math.rutgers.edu/~zeilberg/Opinion104.html).

Re: Keeping computers from ending science's reproducibility

#22

Post by stevenbedrick »

sparky wrote: post #6
jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

To echo the meme around here, there appears to be an opportunity here for a startup :) If you can get buy-in from the research community, even in one niche field, I could see great utility in a central escrow house of sorts, where people can post research software, datasets, etc. and others can download the stuff and validate the results only after agreeing in some reasonably binding way not to use that information t…

In biomedicine, the National Library of Medicine has been taking steps in this direction. There are repositories for sequence, protein structure, and microarray data, for example, and many journals require that relevant data be deposited in these repositories as a condition of publication.

Efforts such as the Science Commons are also thinking about/working towards these sorts of solutions.

Re: Keeping computers from ending science's reproducibility

#23

Post by jedbrown »

yummyfajitas wrote: post #20
jedbrown wrote: post #18

Earlier quoted context omitted.

I agree with many of your sentiments. Here is a view from "inside" (I do PDE solvers from a mathematics and computational physics perspective, have written open-source models, and contribute to a popular open-source solver library). 1. Very little of numerical analysis is about propagation of rounding errors. I think you underestimate the reliability of floating point numbers in this context. The challenge of numeric…

4. It is often not worth the effort to turn a one-off code into distributable software. There is a good chance that only a small number of people care about the PDE you are trying to solve. I contemplated making a distributable solver for the Schrodinger equation, but there was little interest. A quote I've heard attributed to Cleve Moler (chief scientist at Matlab): "PDEs are a niche market."

I think that quote holds less and less true over the years. It's not so much that there are huge numbers of people who need their PDEs solved, rather that those who do (aerospace, engines, circuits, oil, climate, defense, energy) really need them solved and are willing to pay for it.

One thing I enjoy about PDE solvers is that algorithmic improvements always pay off because every user is running the largest simulation they can on the most expensive hardware they can afford.

Re: Keeping computers from ending science's reproducibility

#24

Post by btilly »

yummyfajitas wrote: post #5
jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

I am often told that I should keep my nose out of other science domain's business because they know more than I do. I've only heard such statements coming out of a few fields: math education, labor economics, climate science and psychometrics of race/gender. You should ignore such statements; they are nothing more than an attempt to bully you into accepting received wisdom from activists with a PhD. As an actual scie…

Such statements also come out of biologists discussing evolution. This is not, however, evidence that they aren't really doing science. Instead it is evidence that they've been burned out explaining basics over and over again to Creationists and want to get on with their lives.

However some do take the energy out for those explanations. One of the results of their energy is http://www.talkorigins.org/.

Hopefully some day someone will take the energy to do the same with climate science. Because as much as there is a lot of politics there, there is some real science there as well.

Re: Keeping computers from ending science's reproducibility

#25

Post by Retric »

yummyfajitas wrote: post #5
jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

I am often told that I should keep my nose out of other science domain's business because they know more than I do. I've only heard such statements coming out of a few fields: math education, labor economics, climate science and psychometrics of race/gender. You should ignore such statements; they are nothing more than an attempt to bully you into accepting received wisdom from activists with a PhD. As an actual scie…

Education is a lot more complex than most people understand it to be. Even within the field few people actually have any understanding of how the brain and learning interact. Everything from diet, to time of instruction, to setting, to materials, to methods, to testing is all important. A classic example of which is how do you build a useful test? Well if you want to know how much learning took place you need to test them both before and after instruction. But you also need to test people twice without instruction to see if the tests give information away. You need a mix of problem difficulty to find out if someone understands the basics and at the same time, to see if they understood the minor details. etc. You then need to look at the test not just in terms of overall score but also how well they did on each type of problem.

PS: Education might seem like a flaky field and much of the discussions are devoid of good science. At the same time there is a lot of great research that has been which has real and important implications.

Re: Keeping computers from ending science's reproducibility

#26

Post by carbocation »

sparky wrote: post #6
jerf wrote: post #2

I am often told that I should keep my nose out of other science domain's business because they know more than I do. However, I think when they start building their science on top of computers, I start getting a say again. Here's what concerns me about this increasing use of computers: It seems like the vast bulk of these simulations are iterative, and therefore subject to mathematical chaos. How many of these researc…

To echo the meme around here, there appears to be an opportunity here for a startup :) If you can get buy-in from the research community, even in one niche field, I could see great utility in a central escrow house of sorts, where people can post research software, datasets, etc. and others can download the stuff and validate the results only after agreeing in some reasonably binding way not to use that information t…

Actually, the NIH has already done this with many recent grants. Everyone funded through certain grants has to make their data available in dbGAP (genotype and phenotype), sometimes even pre-publication (which is raising some feathers).

Point 1 will always be true when people share methods and/or data pre-publication. If point 2 cannot be discerned with currently available methods (and I would argue that it can), then we are in a world of trouble.

Re: Keeping computers from ending science's reproducibility

#27

Post by deckard »

plinkplonk wrote: post #14
deckard wrote: post #12

Here's a practical solution I have proposed in my community (autonomous robots): 1. package code and data into tarball or VCS repo.; 2. place package on long-lived website; 3. compute SHA1 hash or similar from package (if git is used, this is the revision ID, conveniently); 4. publish the URI and hash in any paper that makes claims based on that code or data; 5. as a reviewer, prefer papers that follow this method, a…

"My group does this with every paper." Any links?

Sure. We started doing this last summer, and this is the first paper accepted for publication that uses the method.

http://autonomy.cs.sfu.ca/doc/wawerla_icra10.pdf

and here's a draft of our paper on the methodology, with rationale.

Long paper: http://autonomy.cs.sfu.ca/doc/wawerla_submitted_2009.pdf

Original short workshop paper:

http://autonomy.cs.sfu.ca/doc/wawerla_rss09_workshop.pdf

For context, here's the lab's publication list:

http://autonomy.cs.sfu.ca/publications.html

We plan to continue this, even if the idea doesn't catch on.

Re: Keeping computers from ending science's reproducibility

#28

Post by Retric »

jerf wrote: post #13
mattheww wrote: post #7

Earlier quoted context omitted.

Your concerns are certainly legitimate, and I'm sure there are some situations where they apply. However, in many physical sciences (I would venture to say the vast majority), simulations are not chaotic. They are based on stochastic models that have converging behavior. So, in general, it really is fair for most people to rely on the stability of the simulations. As for your other points, the issue of simulation acc…

"As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used.... my field (high energy physics)" Bad example, inasmuch as it is too good . Particle physics has petabytes of data (exabytes yet?) to test against and is very connected to the real world. With that check you can't stray very far. "Ideally, you should collect your own data, make your own s…

Weather simulations have even more data; they are less accurate for other reasons.

Also, running someone else’s exact simulation again is useless. You need to start from scratch (or some vary well accepted baseline libraries) for it to be useful.

Re: Keeping computers from ending science's reproducibility

#29

Post by lionhearted »

btilly wrote: post #24
yummyfajitas wrote: post #5

Earlier quoted context omitted.

I am often told that I should keep my nose out of other science domain's business because they know more than I do. I've only heard such statements coming out of a few fields: math education, labor economics, climate science and psychometrics of race/gender. You should ignore such statements; they are nothing more than an attempt to bully you into accepting received wisdom from activists with a PhD. As an actual scie…

Such statements also come out of biologists discussing evolution. This is not, however, evidence that they aren't really doing science. Instead it is evidence that they've been burned out explaining basics over and over again to Creationists and want to get on with their lives. However some do take the energy out for those explanations. One of the results of their energy is http://www.talkorigins.org/ . Hopefully som…

> Such statements also come out of biologists discussing evolution.

A lot of my friends are scientists, including in a couple biochemists and some other people that do more or less serious research into topics like that. In my experience, you're off on how they deal with stupid people and stupid arguments - instead of "get out of biology" and moving on with their lives, they tend to address and correct errors, debate if necessary, or at least point the person towards a relevant piece that explains things and take new criticisms and arguments and address them if necessary.

The scientists I know always been open to me saying stupid things (and occasionally not stupid things) and correcting me if I'm wrong, or exploring together if I might not be wrong. Good biologists don't say, "Get out of biology".

> Hopefully some day someone will take the energy to do the same with climate science.

Oh, hah, I honestly responded quickly and missed the climate science analogy originally. If you wanted to make an apologia for climate science, then I do understand why you'd want to draw a biology:evolution:creationism to climate science:global warming:deniers parallel.

I was responding to the "want to get on with their lives" part as being wrong based on my experience, as scientists are usually rather encouraging and tolerant of dissent. Climate science is not so much interested in people and data which disagree with them, which is a pretty big problem.

Re: Keeping computers from ending science's reproducibility

#30

Post by frossie »

andrewcooke wrote: post #10

i had this problem many years ago (15?). at the time i was working as a postdoc, calculating the evolution of the ionizing background with redhshift from the inverse effect (lyman alpha clouds near quasars get fried by the quasar; the extent of this gives an indirect way to measure the ionizing background at that redshift). i had a bunch of perl scripts (ah, those were the days) that mangled various files before feed…

But this is a problem that can be tackled, and those who take it seriouslt already do so. For example, in our pipelines we use an infrastructure that always adds every command executed on the file, with every exact parameter, to the metadata of the file, starting from one canonical archived file - and hence, one can indeed reproduce manually the result of the pipeline given sufficient time and dedication. [Edit: we also write the git shar1]

The same way that say, biology labs have processes they engage in in order to convince us that their samples are not contaminated, we can have processes that lead to the reproducibility of data.

Post reply on HN