Live data from Hacker News

Simple rules for documenting scientific software

journals.plos.org

11–20 of 55 posts

Re: Simple rules for documenting scientific software

#11
post #8
post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

This is both wrong and dangerous. I do hear this sentiment often: "Hey dudes, it's just a prototype! Why waste time making it pretty? Who cares? We're doing SCIENCE here!" But prototypes (and beginnings in general) are precisely the time to be extra careful. Wrong turns and self-delusion are costlier, not cheaper, when you're the one paving the road for others. In research, there are many ways to lead yourself astray…

There is a wide difference between code that is pretty and code that is careful and written in a way that will not lead you astray. And if you don't know or don't care to make that distinction I am not interested in your opinion.

Re: Simple rules for documenting scientific software

#12
post #9

Earlier quoted context omitted.

I write scientific code for a living and my code is very difficult for anyone else to maintain. It is not because my code is badly documented or written, it is because what it does is very complex. Every module is documented why it exists and what it does and the code straightforward to read, yet the interaction of all the modules is very complex as it reflects the underlying complexity of the problem the code is sol…

Managing complexity is one of the chief concerns of software engineering.

Complexity can be managed, but the required domain knowledge is hard to design around. The problem my colleagues struggle with is the domain knowledge I have that they don't. The positive from this is there are few people in the world that can do my job :)

Re: Simple rules for documenting scientific software

#14
post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

I'll just put this here:

https://news.ycombinator.com/item?id=18368455

This is normal. Poor sw practices lead to bugs, which lead to invalid science.

Also

https://physicstoday.scitation.org/do/10.1063/PT.6.1.2018082...

This got caught because it led to a big fight. Most other scientific software is not audited at all. Given my experience with those who write them, I tend not to trust most conjugational results coming from science.

Re: Simple rules for documenting scientific software

#15
post #14
post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

I'll just put this here: https://news.ycombinator.com/item?id=18368455 This is normal. Poor sw practices lead to bugs, which lead to invalid science. Also https://physicstoday.scitation.org/do/10.1063/PT.6.1.2018082... This got caught because it led to a big fight. Most other scientific software is not audited at all. Given my experience with those who write them, I tend not to trust most conjugational results coming…

Oh it is nothing new that HN is bashing code in science. But if anything that story is a reason for each group to build their own independent code instead of trying to enforce so much software engineering overhead until there is only one code by the best funded group is left because nobody else dares to (or can afford to) build their own code.

Re: Simple rules for documenting scientific software

#16
post #8

Earlier quoted context omitted.

This is both wrong and dangerous. I do hear this sentiment often: "Hey dudes, it's just a prototype! Why waste time making it pretty? Who cares? We're doing SCIENCE here!" But prototypes (and beginnings in general) are precisely the time to be extra careful. Wrong turns and self-delusion are costlier, not cheaper, when you're the one paving the road for others. In research, there are many ways to lead yourself astray…

There is a wide difference between code that is pretty and code that is careful and written in a way that will not lead you astray. And if you don't know or don't care to make that distinction I am not interested in your opinion.

> Articulating your thoughts into a sane logical structure (aka code), with sane names and motivation examples and conceptual units, saves you time even in the short run.

If you think he or she is only talking about pretty code, you don't understand the distinction to begin with.

Re: Simple rules for documenting scientific software

#17
post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

These incentives exist but are not a good excuse in my view. I'd recommend reading this blog post about how "incentives" are used to justify all sorts of bad behavior in academia: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...

Right now I'm nearly done a PhD in mechanical engineering. I've worked on several computational projects of varying levels of code quality from good (e.g., NIST's Fire Dynamics Simulator: https://github.com/firemodels/fds) to bad (won't give unambiguous examples for obvious reasons...).

I see clear incentives against uncertainty quantification and high quality model validation in my field. Those topics are treated superficially if at all, because people just want their model to look good, and if they do a rigorous job, that increases the chance that their model looks bad. As far as I'm concerned, if you don't do good quality model validation, you're not doing your job as a scientist.

A more specific example: Some codes are kept for a decade or longer if the software is sufficiently complex. One code I worked with was used by multiple graduate students over several PhDs and had very little if anything in terms of documentation. What did exist was far out of date. I spent a fair amount of time trying to understand and document the software, but other people treated me as if I was crazy for wanting to understand the black box I was provided. I don't think I ever actually understood that software fully despite the many hours I spent working on it. And I'm not certain that it provided correct results either. There were almost no tests, and the existing tests were hard to run and were consequently almost never run. The software produced several PhDs, though. The first few PhDs had it fairly easy. I had to pay their technical debt, and I couldn't pay it all back. It made me look bad as far as I'm concerned and did not save time in the long run. If anything, there's a long term incentive toward good software development practices.

Ultimately I think the solution would involve having minimum standard for scientific software quality, or else grant funding will be withdrawn. It will anger many people who have so far gotten away with writing low quality software, but beyond that I don't see many downsides.

Re: Simple rules for documenting scientific software

#18
post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

In industry, we often also face an incredible amount of uncertainty in the form of unknown future functionality, UX, requirements, etc. As long as a business continues to adapt to changing market conditions, it's rare for its software to reach completeness. Some components' rate of change certainly slows down over time but newer components might rapidly grow in complexity or face major refactors. Successfully scaling to meet increased demand may also demand a pretty fundamental architectural shift. While not entirely the same, your evolving understanding of a problem and its effect on software requirements isn't too different from what we see regularly in industry.

It's very difficult to foresee what changes will be required 1-year's time, let alone 3. To that end, devs try to be extra thoughtful during the planning phase so that the end product is made up of well-behaved components. The extra time taken to come up with a sensible architecture goes a long way when business requirements eventually call for new or enhanced functionality and someone needs to understand the codebase in a reasonable amount of time. To be clear, this doesn't guarantee clean code -- bad architectural decisions happen, planning can occur without a clear requirements, no one held the quality bar during implementation, etc. -- but the outcome is always better than a laissez-faire design "strategy".

Perhaps scientific code churn is much more frequent than commercial code churn. In which case you would want to balance that by perhaps spending less time coming up with a good architecture -- and in the case of throwaway code, perhaps none at all. That being said, I would be surprised if new scientific knowledge always results in you having to discard every line and restart completely from scratch. I imagine that spending more time upfront architecting your software would instead allow you to more quickly react to new research results -- paying dividends on your time, in the long run.

Re: Simple rules for documenting scientific software

#19
post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

These incentives exist but are not a good excuse in my view. I'd recommend reading this blog post about how "incentives" are used to justify all sorts of bad behavior in academia: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc... Right now I'm nearly done a PhD in mechanical engineering. I've worked on several computational projects of varying levels of code quality from good (e.g., NIST's Fire Dynamics…

I agree that not enough benchmarking (comparision between different codes to check that they get compatible results), verification (checking that the code produces results consistent with the underlying model) and validation (checking that the code reproduces nature to a sufficient degree to be useful) it done.

I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. But unlike you and a lot of other posters here on HN I don't think the reason for insufficient validation is that we scientists are too stupid or too lazy. On the contrary, we would love to do more of it. But there is next to no funding to do it.

Re: Simple rules for documenting scientific software

#20
post #6

If there is one thing I have about HN it is how smug they are about software engineering. The leading reason why scientific software stinks (and a fair fraction does) is not because scientists and software engineeres in that field suck, but because there is very strong incentives AGAINST writing better software. Remember: This is not the 17th javascript framework, but software for problems that we don't understand go…

By this argument, chemists should never wash glassware. Keep re-using it until it's too dirty, then through it out and make new beakers.

Your tools affect the quality of your work. Work with shitty tools, get shitty results.

I'll start believing your argument when scientists actually start publishing the code to go with their papers and make it reproducible.

Post reply on HN