Simple rules for documenting scientific software • Hacker News Board

Re: Simple rules for documenting scientific software

#21

Post by btrettel » Sun, Dec 23, 2018, 12:07 AM UTC

BeetleB wrote: post #14

petschge wrote: post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

I'll just put this here: https://news.ycombinator.com/item?id=18368455 This is normal. Poor sw practices lead to bugs, which lead to invalid science. Also https://physicstoday.scitation.org/do/10.1063/PT.6.1.2018082... This got caught because it led to a big fight. Most other scientific software is not audited at all. Given my experience with those who write them, I tend not to trust most conjugational results coming…

The second link was fascinating and would make a good submission to HN.

> Given my experience with those who write them, I tend not to trust most conjugational results coming from science.

In computational fluid dynamics, there's a saying: No one believes computations aside from the person who ran them, and everyone believes experiments aside from the person who ran them.

To be fair, this is mostly because turbulence modeling can be very inaccurate, but bugs in the code are also a major concern of mine.

Re: Simple rules for documenting scientific software

#22

Post by btrettel » Sun, Dec 23, 2018, 12:12 AM UTC

petschge wrote: post #19

btrettel wrote: post #17

Earlier quoted context omitted.

These incentives exist but are not a good excuse in my view. I'd recommend reading this blog post about how "incentives" are used to justify all sorts of bad behavior in academia: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc... Right now I'm nearly done a PhD in mechanical engineering. I've worked on several computational projects of varying levels of code quality from good (e.g., NIST's Fire Dynamics…

I agree that not enough benchmarking (comparision between different codes to check that they get compatible results), verification (checking that the code produces results consistent with the underlying model) and validation (checking that the code reproduces nature to a sufficient degree to be useful) it done. I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. But…

I appreciate your thoughts.

> I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them.

I don't know what field you work in, but in my field (fluid dynamics) these papers are not uncommon, though often poorly done. There's a relatively new ASME journal on the subject too: http://verification.asmedigitalcollection.asme.org/journal.a...

> On the contrary, we would love to do more of it. But there is next to no funding to do it.

Perhaps this is the crux of the disagreement then. I don't need extra funding to do what I should be doing in the first place. It can be a big time sink in some cases (e.g., I probably spent 6+ months compiling existing data during my PhD), but often is not. It will probably make your model look worse than you'd like, but if everyone did it then science would progress faster.

Re: Simple rules for documenting scientific software

#23

Post by petschge » Sun, Dec 23, 2018, 12:12 AM UTC

jasonpeacock wrote: post #20

petschge wrote: post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

By this argument, chemists should never wash glassware. Keep re-using it until it's too dirty, then through it out and make new beakers. Your tools affect the quality of your work. Work with shitty tools, get shitty results. I'll start believing your argument when scientists actually start publishing the code to go with their papers and make it reproducible.

Two thirds of the codes in my field are on github. And for most others you can get a copy if you ask politely by email. That said I would appreciate it if journals not only had author, title, date and affiliation in the meta data, but also a git url and commit ID.

Re: Simple rules for documenting scientific software

#24

Post by petschge » Sun, Dec 23, 2018, 12:21 AM UTC

btrettel wrote: post #22

petschge wrote: post #19

Earlier quoted context omitted.

I agree that not enough benchmarking (comparision between different codes to check that they get compatible results), verification (checking that the code produces results consistent with the underlying model) and validation (checking that the code reproduces nature to a sufficient degree to be useful) it done. I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. But…

I appreciate your thoughts. > I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. I don't know what field you work in, but in my field (fluid dynamics) these papers are not uncommon, though often poorly done. There's a relatively new ASME journal on the subject too: http://verification.asmedigitalcollection.asme.org/journal.a... > On the contrary, we would love to d…

Kinetic plasma simulations in space plasma and astrophyscial scenarios. I have seen that journal and I think it is a step in the correct direction.

Regarding the funding: you kind DO need that extra funding. because if you don't get funding for that you get out-competed by the other research who skips that step and the next postdoc position or grant goes to that researcher who wrote two more papers instead of testing code that ultimately didn't find any major bugs.

Re: Simple rules for documenting scientific software

#25

Post by chrisweekly » Sun, Dec 23, 2018, 12:33 AM UTC

Radim wrote: post #8

petschge wrote: post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

This is both wrong and dangerous. I do hear this sentiment often: "Hey dudes, it's just a prototype! Why waste time making it pretty? Who cares? We're doing SCIENCE here!" But prototypes (and beginnings in general) are precisely the time to be extra careful. Wrong turns and self-delusion are costlier, not cheaper, when you're the one paving the road for others. In research, there are many ways to lead yourself astray…

SOTA -- State of the Art?

Re: Simple rules for documenting scientific software

#26

Post by btrettel » Sun, Dec 23, 2018, 12:37 AM UTC

petschge wrote: post #24

btrettel wrote: post #22

Earlier quoted context omitted.

I appreciate your thoughts. > I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. I don't know what field you work in, but in my field (fluid dynamics) these papers are not uncommon, though often poorly done. There's a relatively new ASME journal on the subject too: http://verification.asmedigitalcollection.asme.org/journal.a... > On the contrary, we would love to d…

Kinetic plasma simulations in space plasma and astrophyscial scenarios. I have seen that journal and I think it is a step in the correct direction. Regarding the funding: you kind DO need that extra funding. because if you don't get funding for that you get out-competed by the other research who skips that step and the next postdoc position or grant goes to that researcher who wrote two more papers instead of testing…

> you kind DO need that extra funding. because if you don't get funding for that you get out-competed [...]

This seems to be a common view, but I don't think it's always true. In my own work, I've found that being careful has led me to write more papers, not less. Particularly given that I've found errors in previous work that I intend to publish or already have published. There's a delay involved (i.e., it takes time to find the problems) but I think I'm coming out ahead in total publications in the long run.

Re: Simple rules for documenting scientific software

#27

Post by acidburnNSA » Sun, Dec 23, 2018, 12:47 AM UTC

I manage a team of power plant design engineers writing complex scientific HPC software, mostly in Python (which drives Fortran 77 codes behind the scenes, among other things). It's been a long haul but through the years we've learned a lot of good lessons and are pretty productive at what I consider at least moderately good code. Everyone starts off by reading Clean Code and taking basic proficiency training. We know to try to make code speak for itself because, as we've all seen many times, comments lie. We've slowly learned from the software engineering community how to set up things like Jenkins for CI, black for Python code auto-formatting, Phabricator for revision control policy and code review. We're writing docs in rst with sphinx, watching coverage, and getting pylint 10/10. We're waking up to the wonders of static types.

Mandatory code review has turned out to be a great way to train newbies, write better code, and make sure code is understandable to at least one other person. It's been an investment (it's slowww) but I still think it pays dividends. That's what I'd add to this advice. And I'd tone down the comments focus. Comment only when you fail to express yourself clearly in the code.

Re: Simple rules for documenting scientific software

#28

Post by mturmon » Sun, Dec 23, 2018, 12:48 AM UTC

petschge wrote: post #19

btrettel wrote: post #17

Earlier quoted context omitted.

These incentives exist but are not a good excuse in my view. I'd recommend reading this blog post about how "incentives" are used to justify all sorts of bad behavior in academia: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc... Right now I'm nearly done a PhD in mechanical engineering. I've worked on several computational projects of varying levels of code quality from good (e.g., NIST's Fire Dynamics…

I agree that not enough benchmarking (comparision between different codes to check that they get compatible results), verification (checking that the code produces results consistent with the underlying model) and validation (checking that the code reproduces nature to a sufficient degree to be useful) it done. I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. But…

You are making some good points here. It is expensive and exacting work to, for instance, set up validation experiments, or to quantify errors exactly enough so that whatever errors that do appear are explained by known effects. Agencies don’t like to pay for this.

One forcing function can be if high-stakes decisions rest on the conclusions from the code. In the geosciences, you start to see some of this care for air quality measurements and for climate science. (One relevant journal here is SIAM J. UQ., https://www.siam.org/Publications/Journals/SIAM-ASA-Journal-...)

The lab where I work has spent a lot of resources on validating thermal and fluid dynamical codes for atmospheric entry of Mars landers — an engineering example rather than science. They obviously are very motivated to avoid an unpleasant surprise from a model failure.

Re: Simple rules for documenting scientific software

#29

Post by petschge » Sun, Dec 23, 2018, 12:49 AM UTC

btrettel wrote: post #26

petschge wrote: post #24

Earlier quoted context omitted.

Kinetic plasma simulations in space plasma and astrophyscial scenarios. I have seen that journal and I think it is a step in the correct direction. Regarding the funding: you kind DO need that extra funding. because if you don't get funding for that you get out-competed by the other research who skips that step and the next postdoc position or grant goes to that researcher who wrote two more papers instead of testing…

> you kind DO need that extra funding. because if you don't get funding for that you get out-competed [...] This seems to be a common view, but I don't think it's always true. In my own work, I've found that being careful has led me to write more papers, not less. Particularly given that I've found errors in previous work that I intend to publish or already have published. There's a delay involved (i.e., it takes tim…

Right. You do not always get outcompeted. But you run the risk. And yes, having good architecture in parts you need to change often is someting that pays of. As does having tests in section of the code that is brittle, hard to reason about or historically buggy. As does writing documentation on things that you had to spend an annoying amount of time on to understand. And I think that most researchers understand that and invest that time.

The thing is: This does not mean that either of us puts a lot of effort into having nice, good, modular architecture in parts we know that we will never change. Or write a lot of code coverage in module we understand very well and can reason about using analytic calculations. Or write a lot of documentation for things that are obvious to us (be it through familiarity or whatever reason). The next researcher however might want to change exactly that part we never want to change. Or might the analytic check we always used to make sure to code was not going of the deep end is not valid for his work. And he is near certainly going to need documentation on other spots of the code (and find the documentation we wrote and needed useless, because that topic is something glaringly obvious to him).

There is usually no advantage for us to make the code nice for that researcher. And THAT is very scientific software gets the bad name from. Because we have to make the trade-off, what is useful to us (and possibly our time). If we try to make the code nice and friendly for every potential user under the sun we usually DO get outcompeted.

Re: Simple rules for documenting scientific software

#30

Post by btrettel » Sun, Dec 23, 2018, 1:03 AM UTC

petschge wrote: post #29

btrettel wrote: post #26

Earlier quoted context omitted.

> you kind DO need that extra funding. because if you don't get funding for that you get out-competed [...] This seems to be a common view, but I don't think it's always true. In my own work, I've found that being careful has led me to write more papers, not less. Particularly given that I've found errors in previous work that I intend to publish or already have published. There's a delay involved (i.e., it takes tim…

Right. You do not always get outcompeted. But you run the risk. And yes, having good architecture in parts you need to change often is someting that pays of. As does having tests in section of the code that is brittle, hard to reason about or historically buggy. As does writing documentation on things that you had to spend an annoying amount of time on to understand. And I think that most researchers understand that…

I think I misunderstood your position earlier. My experience with academic codes seems to differ significantly from yours. It is not uncommon at all for scientific software to have no documentation at all in my experience, and very little if anything in terms of tests. I agree with you that it does not make sense to have detailed documentation and tests for all parts of scientific software. One must prioritize. What I'm arguing against is the (often implicit) attitude that documentation, testing, and other good software engineering practices are not necessary in science at all. I see now that you agree, and it's more of a question of the amount of software engineering practices which are optimal. I appear to prefer more than you do, but you clearly see the value of these practices. Correct me if I'm wrong.