Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write better scientific code in Python?

zerowithdot.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: How to write better scientific code in Python?

#31

Post by mattkrause »

kortex wrote: post #5

Excellent to see functional programming ideas like deferred computation and clean interfaces make their way into the scientific computing space. One thing though: I know these are good ideas. But to someone not as familiar with these patterns, they may wonder "why go through all this trouble?"

I thought the deferred part could be better.

For example, one easy win would be to replace the list comprehensions with a generator: there's no point in allocating that entire list just so that statistics.mean can iterate over it.

At the opposite end, the switch to a Distribution class also enables a huge speedup: keep the sampling machinery for situations where you actually need (Cacuhy?), but while die.expected_value() returns (self.n_sides+1)/2, which is effectively free.

Re: How to write better scientific code in Python?

#32

Post by toponaut »

The author's solution to the toy problem is

---

die = Die(12)

expected_values(die, n=10000)

gaussian = Gaussian(mu=4.0, sigma=2.0)

expected_value(gaussian, n=100000)

coin = Coin(fairness=0.75)

expected_value(coin, f=lambda x: np.where(x == "H", 1.0, 0.0)

---

After all this work, the answers to the problems are right there in the constraints.

ev_die = (sides+1)/2

ev_gaussian = mu

ev_coin = sign * fairness

What is the point?

Re: How to write better scientific code in Python?

#34

Post by Someone »

I think the author of this article should read the Mythical Man Month (https://en.wikipedia.org/wiki/The_Mythical_Man-Month), in particular on the difference between (all quotes are from that book)

- a program, “complete in itself, ready to be run by the author on the system on which it was developed”

- a programming product: “a program that can be run, tested, repaired, and extended by anybody. It is usable in many operating environments, for many sets of data”

- a programming system: “a collection of interacting programs, coordinated in function and disciplined in format, so that the assemblage constitutes an entire facility for large tasks“

- a programming systems product: “This differs from the simple program in all of the above ways. […] But it is the truly useful object, the intended product of most system programming effort”

They all have their place, and if you need a program, spending time on writing a product, a programming system or a programming systems product is a waste of effort.

Most scientific code falls in the program or, maybe, somewhat in the direction of the programming product category, and there’s nothing wrong with that.

(Note that quality is a concept that’s orthogonal to this distinction)

Re: How to write better scientific code in Python?

#37

Post by toponaut »

d0mine wrote: post #22
toponaut wrote: post #19

Earlier quoted context omitted.

To a software engineer this will also read mad. This article is ultimately explaining why abstraction is good and why it's helpful to build classes in python. That is already obvious to SWEs, not at all specific to "scientific computing", and explained elsewhere much more succinctly.

I see it more as an example of anti pattern: unnecessary layers obfuscating what happening. Good abstraction is hard and it is not free. It is better to make mistake of under using abstraction than overusing (the latter is much harder to maintain). I would understand if the list comprehension were replaced with the corresponding numpy/scipy/pandas/etc code.

I agree. I didn't mean to imply this was a good explanation or example. I still think "abstraction/classes are good" was the intention/gist of the article.

Re: How to write better scientific code in Python?

#38

Post by jointpdf »

The terrible irony of this post is that they use a completely wrong definition of expected value. They write down an integral...but then implement something totally different.

You cannot calculate the EV of a random variable X by taking the mean of random samples drawn from the distribution of X (e.g. try it with the Cauchy distribution—see if you get something approaching the actual EV). The only thing they accomplished is building up a very convoluted way of calculating a sample mean—which is already trivial with Numpy or just standard Python. Why do this? I’m perplexed.

This is the issue with software engineers writing scientific code—they often flagrantly misunderstand basic mathematical definitions, and then obfuscate this misunderstanding with their “pure” and “robust” code.

Sorry for the harsh criticism, but I’m tired of seeing this kind of thing.

Re: How to write better scientific code in Python?

#39

Post by voorwerpjes »

Former academic scientist, now software engineer.

I think this article misses the point that rarely in science is the code the actual product/deliverable. The way a scientific code should be judged is it performant enough to answer my question in a reasonable amount of time, clear enough to another reader (which face it 99% of the time, in science the only other reader will likely be your future self or a student with little coding experience), and does it give the correct result in a reproducible way. To my eye, the first "bad" example is actually the best to all these questions.

To my mind it feels more like how can I use my scientific coding I have to do as a way to improve my software engineering skills. Which isn't a bad endeavor and might make leaving academic science easier for you, heck I did the same thing. However, in my humble opinion, it won't make you write "better" scientific code.

Re: How to write better scientific code in Python?

#40

Post by bmitc »

mjburgess wrote: post #14

I think to any audience other than fairly hardcore software engineers this is going to read a little... mad. The first code example was exceptionally clear, from then on, we get increasingly incomprehensible. "Better" it isnt. Scientific code is often write-once, run-once, done. And since mathematics/statistics is largely already formalised at the level of distributions, sampling, and so on -- we shouldn't expect to…

> Scientific code is often write-once, run-once, done.

I don't buy that in the sense that you mean. In my experience, code generally lives on no matter what, and code that remains write-once is often because no one can read it and thus update it (i.e., write it again). I've seen a decent amount of scientific code, and although it was never meant to be ran more than once by anyone other than the original author, it almost always was done so. And because it wasn't designed well, written well, or even commented well, code like that basically becomes this immutable artifact that people simultaneously don't want touched but also want it updated and keep on running.

I did an REU in mathematics back in the day where I wrote a decent amount of MAPLE code. I wasn't even a software engineer at the time, since I was a math and EE major. I honestly had little concept of what computer science even was and was not a programmer. It was not hard at all to write decent code that was commented and organized.

The problem is that most scientists, in my experience, view software and code as somewhat below them. It's like a hammer and a box of nails to them.

I inherited someone's Ph.D. dissertation Python code that had lived on across several labs. It had basically reached the "needs a rewrite" state because it was impossible to get it to run. It was stuck on Python <2.7 and basically couldn't be upgraded without replacing a lot of it because the Python distribution it was based on had become deprecated and none of the packages could be upgraded without changing a lot of the code. I'm not sure I ever saw a single comment. The person who originally wrote the code actually ended up working at one of the major scientific Python shops, somewhere I once applied to and was unceremoniously turned down. An interesting turn of events.

Post reply on HN