Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write better scientific code in Python?

zerowithdot.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: How to write better scientific code in Python?

#21

Post by atoav »

mjburgess wrote: post #14

I think to any audience other than fairly hardcore software engineers this is going to read a little... mad. The first code example was exceptionally clear, from then on, we get increasingly incomprehensible. "Better" it isnt. Scientific code is often write-once, run-once, done. And since mathematics/statistics is largely already formalised at the level of distributions, sampling, and so on -- we shouldn't expect to…

I think some degree of abstraction can lead to more clarity – especially if it hides a bunch of crap, that is done over and over again throughout the code. But these abstractions need to be very well chosen, clearly named and unless they really give you more clarity (e.g. by hiding some confusing details to bring the general point you are trying to make across more clearly) I would advice against them.

If your scientific code is meant to be used in actual software or if you want to write tests for the functions used, a little bit more abstraction might be a good idea, however.

Re: How to write better scientific code in Python?

#22

Post by d0mine »

toponaut wrote: post #19
mjburgess wrote: post #14

I think to any audience other than fairly hardcore software engineers this is going to read a little... mad. The first code example was exceptionally clear, from then on, we get increasingly incomprehensible. "Better" it isnt. Scientific code is often write-once, run-once, done. And since mathematics/statistics is largely already formalised at the level of distributions, sampling, and so on -- we shouldn't expect to…

To a software engineer this will also read mad. This article is ultimately explaining why abstraction is good and why it's helpful to build classes in python. That is already obvious to SWEs, not at all specific to "scientific computing", and explained elsewhere much more succinctly.

I see it more as an example of anti pattern: unnecessary layers obfuscating what happening. Good abstraction is hard and it is not free. It is better to make mistake of under using abstraction than overusing (the latter is much harder to maintain).

I would understand if the list comprehension were replaced with the corresponding numpy/scipy/pandas/etc code.

Re: How to write better scientific code in Python?

#23

Post by bluenose69 »

This article starts with naive code that is unlikely to be written by working scientists unless they are really quite new this type of work.

And, quite quickly, the article reaches a level of detail that might not engage the initial readers.

As a scientist with several decades of programming experience, it seems that those who write this sort of naive code are at an early stage of learning, and they might be better off learning Julia, which offers high performance even if written in a naive style. Granted, Python has superior (formal and informal) documentation, and it's error message are a lot easier to understand than those spewed by Julia. But it is quite freeing to be able to express things in a manner that is closer to mathematical notation, without paying a high performance cost. And, if the task really requires it, Julia will let you tune things to Fortran-level speeds (and sometimes better ... but that's another story).

Re: How to write better scientific code in Python?

#24

Post by pphysch »

On a related note I recently wrote some scientific codes with Go. MPI bindings were ergonomic enough and generics is a big win for writing reusable & fairly performant codes... Built once and deployed on multiple nodes without a hitch. And zero mucking with venvs.

I feel like if you're reaching for interfaces and type hints in Python, it's time to revisit the ole toolkit.

Re: How to write better scientific code in Python?

#25

Post by sega_sai »

Scientist here, who has been writing code for > 20 years. I don't buy pretty much anything in the article. This was a bunch of opinionated examples. Science programming in my opinion has many different levels which require different approaches and techniques.

Very often the first code written will be just a quick and dirty. If the idea worked out, I may refactor it, make the code prettier, speed it up a bit. Very-very rarely the bit of code will be something that I'll need to constantly reuse, and there I have to think about interfaces, organisation etc. Also a situation that often comes up, you have a working code that does the job, and then you keep adding more and more functionality to it, and that slowly requires making the code more generic, better organized. But this is not the first thing you do, you only do that if it is needed. That's why I think there are very few very generic recommendations for scientific programming.

Re: How to write better scientific code in Python?

#26

Post by nooorofe »

I don't like the code. My start point would be that

    import pandas as pd
    import numpy as np

    if __name__ == "__main__":
        n_samples = 10000
        samples_np = pd.DataFrame(np.random.randint(1, 7, n_samples), columns=["face_value"])
        print(samples_np.face_value.mean())
Speaking about abstraction, I don't know math, so first thought would be to look for *existing* abstractions. When I work with relational data, my first option to check is SQL. For math looks like DataFrame is a *standard* abstraction. To be fair, maybe first I would be using build-in `random.randin` I am not very familiar with `numpy`, but I would definitely google "pandas random sample", that would bring https://pandas.pydata.org/docs/reference/api/pandas.DataFram...

    if __name__ == "__main__":
        n_samples = 10000
        sample_pd = pd.DataFrame({'face_value': [1, 2, 3, 4, 5, 6]})
        print(sample_pd.sample(
            n=n_samples, 
            replace=True, 
            random_state=np.random.bit_generator.randbits(20)).face_value.mean())
code uses lambda functions in some examples, it probably kills advantages of `numpy` performance. Using DataFrame API at least helps to avoid those pitfalls.

Type annotation, I like the idea, but in the end code looks like Java, but doesn't performs like Java. It is very hard to make it right in Python, also some of them wrong.

( @dataclass(frozen=True): - don't need ":" Gaussian.sample - missing return )

when return added it doesn't return `-> Sequence[float]:`

    Gaussian().sample(90).dtype
    >>> dtype('float64')
-> Sequence[Union(numpy.float64, numpy.float32, numpy.float16)]: # ?

I don't believe "scientific code" is fundamentally different from any other code, I would go with following normal development practices

1) review design ("don't reinvent wheel")

2) add tests

3) make code review

4) version control

etc.

Re: How to write better scientific code in Python?

#27

Post by pgorczak »

mjburgess wrote: post #14

I think to any audience other than fairly hardcore software engineers this is going to read a little... mad. The first code example was exceptionally clear, from then on, we get increasingly incomprehensible. "Better" it isnt. Scientific code is often write-once, run-once, done. And since mathematics/statistics is largely already formalised at the level of distributions, sampling, and so on -- we shouldn't expect to…

I get the author’s idea and I’ve been there before. When you’re still exploring a problem domain it’s really useful to abstract and compose because it helps you both understand the problem better and iterate more efficiently.

At some point you’ll want to come back to code that looks more like the first example, ideally with plenty of documentation that explains how you got there. Some other comment put it really well: you create a kind of personal mapping of scientific concepts to programming abstractions. You don’t want to have colleagues (or your future self) figure out that mapping to understand your results in terms of the science you’re working on.

Re: How to write better scientific code in Python?

#28

Post by AitchEmArsey »

This is an article about how to write theoretically better code (from a CS perspective) for a scientific use-case - but that is not the same thing as "better scientific code". For most purposes, the effort a scientist will go to in understanding how classes work (as an example) will outweigh any benefit they might see from using them.

One of the nice things about Python is that the programming style arrived at here is totally unnecessary - a point which was soundly missed by the author.

Re: How to write better scientific code in Python?

#30

Post by noobermin »

mjburgess wrote: post #14

I think to any audience other than fairly hardcore software engineers this is going to read a little... mad. The first code example was exceptionally clear, from then on, we get increasingly incomprehensible. "Better" it isnt. Scientific code is often write-once, run-once, done. And since mathematics/statistics is largely already formalised at the level of distributions, sampling, and so on -- we shouldn't expect to…

Thanks for the level headed critique. I indeed began reading the article in earnest and the moment he started making ABCs I skipped to skimming and came here and was about to rant. This is the right perspective, people need to understand their audience first when writing and at the very least justify why the level of abstracts they employed are needed vs for example, their first suggestion for improving the die example which is what I'd go with, even better, you have numpy.average which would be even more clear for scientific programmers because they already know numpy has arithmetic average built-in and will recognize it.
Post reply on HN