Why scientific programming does not compute • Hacker News Board

Re: Why scientific programming does not compute

#61

Post by notarealname » Wed, Jul 06, 2011, 9:47 PM UTC

[New account for anonymity]

An often neglected force in this argument is that many practitioners of "scientific coding" take rapid iteration to its illogical and deleterious conclusion.

I'm often lightly chastised for my tendencies to write maintainable, documented, reusable code. People laugh guiltily when I ask them to try checking out an svn repository, let alone cloning a git repo. It's certain that in my field (ECE and CS) some people are very adamant about clean coding conventions, and we're definitely able to make an impact bringing people to use more high level languages and better documentation practices.

But that doesn't mean an hour goes by without seeing results reverse due to a bug buried deep into 10k lines of undocumented C or Perl or MATLAB full of single letter variables and negligible modularity.

Re: Why scientific programming does not compute

#62

Post by madaxe » Wed, Jul 06, 2011, 9:48 PM UTC

szany wrote: post #7

gte910h wrote: post #6

I want to write a "software style guide" for journalists and their editors. Software and Code are both mass nouns in technical language. "Code" can be in programs (aka, things that run), libraries (things that other programmers can use to make programs), or in samples to show people how to do things in their programs or libraries. Some people call short programs scripts. When you feel you should pluralize "software",…

Actually it's a science thing. In a scientific context "code" is understood to mean "program". For example: http://scholar.google.com/scholar?q=%22population+synthesis+... I'm not sure why this is though.

Because most scientists write code in Fortran '77, which is interpreted, meaning that "code" == "program".

Re: Why scientific programming does not compute

#63

Post by crocowhile » Wed, Jul 06, 2011, 9:49 PM UTC

earl wrote: post #30

MasterScrat wrote: post #15

Earlier quoted context omitted.

Similarly, from Zed Shaw ( http://learnpythonthehardway.org/book/advice.html ) : "People who can code in the world of technology companies are a dime a dozen and get no respect. People who can code in biology, medicine, government, sociology, physics, history, and mathematics are respected and can do amazing things to advance those disciplines."

My experience is while they may be respected, they aren't paid . And I'm not a charity.

Nobody is paid for doing research. The salary of a professor is about 1/3-1/5 of what the same person may get in industry. People who do research don't do it for the money.

Re: Why scientific programming does not compute

#64

Post by scott_s » Wed, Jul 06, 2011, 9:49 PM UTC

gallamine wrote: post #52

scott_s wrote: post #12

Earlier quoted context omitted.

The point is that since software development is not their main goal or background, their practices tend to be ad-hoc. We know the value of testing and documentation, but they do not. People don't know to stop doing something until they know it's a bad practice. And they're not going to know it's a bad practice until they discover that fact on their own (which can be a slow process) or someone teaches them (faster, bu…

The mindset of a scientist is that the code is a one-time thing to achieve a separate goal - data for a paper. The code isn't supposed to last, it's simply a stepping stone. For a lot of folks, whose research areas tend to move around, there isn't always the expectation that you'll get to a 2nd or 3rd paper on the same data. Now, all of this is different if you research actually is building the model. But, I'm speaki…

It may not be intended to last, but it's still supposed to be correct. And, of course, there's probably gobs of software out there was not intended to last, yet did.

Re: Why scientific programming does not compute

#65

Post by gallamine » Wed, Jul 06, 2011, 9:50 PM UTC

pwang wrote: post #55

gallamine wrote: post #44

I'm a PhD student in Electrical Engineering. I'm currently working on a Monte Carlo-type simulation for looking at the underwater light field for underwater optical communication (no sharks!). I'm doing the development in MATLAB and I recently put all my code up on Github ( https://github.com/gallamine/Photonator ) to help avoid some of these problems (lack of transparency). Even if nobody ever looks/uses the code, I…

There is a market, and it's called libraries. Eventually you will use a language where software carpentry and code reuse is a core feature, and tested, modular libraries for not only core algorithms, but also deployment and dev-ops stuff (like managing a compute cluster on the cloud) will have standard approaches. This is starting to shape up on the Python side of things, but it has stagnated a little bit. People who…

"Eventually you will use a language where software carpentry and code reuse is a core feature."

Well, to some extent products like MATLAB solve this problem. For better or worse, I trust Matlab's ability to generate a (pseudo) random number, parallel process my functions, invert matrices, etc., etc.

On a broader level, thanks to the specialization of academia, chances are that the code I want to write isn't duplicated by others. Even if it is, I still have to trust them to have written it well - which is the whole problem here.

I guess I don't have as much hope as you do.

Re: Why scientific programming does not compute

#66

Post by pwang » Wed, Jul 06, 2011, 9:50 PM UTC

szany wrote: post #7

gte910h wrote: post #6

I want to write a "software style guide" for journalists and their editors. Software and Code are both mass nouns in technical language. "Code" can be in programs (aka, things that run), libraries (things that other programmers can use to make programs), or in samples to show people how to do things in their programs or libraries. Some people call short programs scripts. When you feel you should pluralize "software",…

Actually it's a science thing. In a scientific context "code" is understood to mean "program". For example: http://scholar.google.com/scholar?q=%22population+synthesis+... I'm not sure why this is though.

Yes, I hear this all the time. I know that the fancy course 6 kids here on HN would poo-poo it, but it's a very common usage in scientific computing. I can imagine the origins and can hypothesize about why it persists (the festering petri dishes of programming culture that is "grad school"), but don't have a definitive answer.

John Tukey is widely credited with coining the term "software" in print in 1958, but I'll wager that "codes" actually predates that.

Re: Why scientific programming does not compute

#67

Post by ak217 » Wed, Jul 06, 2011, 9:51 PM UTC

mclin wrote: post #41

Rather than building these data analysis/visualization programs from scratch each time, my thought is that scientists should instead be writing them as modules for a data workflow application like RapidMiner. If you haven't heard of RapidMiner, you basically edit a flowchart where each step takes inputs and outputs, eg take some data and make a histogram, or perform a clustering analysis. Video of someone demoing it:…

In bioinformatics, Galaxy (http://galaxy.psu.edu/) is a much better alternative.

Re: Why scientific programming does not compute

#68

Post by pama » Wed, Jul 06, 2011, 9:51 PM UTC

pwang wrote: post #57

ANH wrote: post #45

From personal experience, I attest that it can be more difficult than pulling teeth to get a scientist to commit code to a version control system.

Greg Wilson once commented that the subversive way to get scientists to use source control was not to pitch it as a code history tool, but rather as a nifty way to sync up code between their work machines, home machines, etc. He said he had a lot more traction with that than trying to lecture them about having code history.

Dropbox has invalidated this pitch.

Re: Why scientific programming does not compute

#69

Post by gallamine » Wed, Jul 06, 2011, 9:51 PM UTC

pwang wrote: post #57

ANH wrote: post #45

From personal experience, I attest that it can be more difficult than pulling teeth to get a scientist to commit code to a version control system.

Greg Wilson once commented that the subversive way to get scientists to use source control was not to pitch it as a code history tool, but rather as a nifty way to sync up code between their work machines, home machines, etc. He said he had a lot more traction with that than trying to lecture them about having code history.

I'm guessing Dropbox has introduced many a scientist to the wonders of code history/portability. I was pretty reluctant to move to Git when Dropbox worked fine.

Re: Why scientific programming does not compute

#70

Post by andrewcooke » Wed, Jul 06, 2011, 9:54 PM UTC

jzila wrote: post #33

My girlfriend is a PhD student in a pharmacology lab. I'm a software engineer working for an industry leader. Once, she and the lab tech were having issues with their analysis program for a set of data. It was producing errors randomly for certain inputs, and the data "looked wrong" when it didn't throw an error. I came with her to the lab on a Saturday and looked through the spaghetti code for about 20 minutes. Once…

i'm in a similar position to you (although i started out as an academic i've worked in the software industry for ages and so end up helping my astronomer partner).

anyway, i disagree slightly with your analysis. in my experience academics know that they suck at the "engineering" part and, to make up for it, are very diligent in making sure that the results "feel right". so i don't think what you described was luck - that's how they work.

in comparison, what drives me crazy, is that if they learnt to use a few basic tools (scm, libraries, an ide, simple test framework) they could save so much time and frustration.

[related anecdote: last year i rewrote some c code written by a grad student that was taking about 24 hours to run. my python translation finished in 15 minutes and gave the same answer each time it was run (something of a novelty, apparently)].