Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

71–80 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#71

Post by AlphaSite »

srean wrote: post #51
lmm wrote: post #48

But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of c…

Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI. I consider it an accident of history that Numpy, Scipy, Pandas, Scikits…

I think Swift's seamless bridging is pretty damn good as well.

Re: Why I Still Use Python for High Performance Scientific Computing

#72

Post by astrafoo »

AlphaSite wrote: post #71
srean wrote: post #51

Earlier quoted context omitted.

Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI. I consider it an accident of history that Numpy, Scipy, Pandas, Scikits…

I think Swift's seamless bridging is pretty damn good as well.

A bridge is always easier to build if there is no river in between.

Re: Why I Still Use Python for High Performance Scientific Computing

#73

Post by kitd »

Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have very different requirements so a combination of a tool that can do all the data handling easily (ie Python) + a tool that can throw the CPU at a numerical problem (ie C) will work as a great combination.

In contrast, if you work in Java, you are trying to use the same tool for both jobs, and you may well fall between 2 stools. And I say that as a typical Java-head.

My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node?

As for the number crunching, I think Rust would be a better choice here. Good memory management is its USP and that can have significant performance benefits.

Re: Why I Still Use Python for High Performance Scientific Computing

#75

Post by megalodon »

boulos wrote: post #18

If Numpy, Pandas, etc. were wrappable from JavaScript this could have easily been titled "Why I use Node.js for High Performance Scientific Computing". The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!". I agree some of the tooling and niceties are beyond a doubt best in breed…

I'm currently working on bringing BLAS [1] and LAPACK [2] to JavaScript. And here is a matrix/vector library with some of the BLAS functions implemented [3].

[1] https://github.com/mateogianolio/nblas

[2] https://github.com/mateogianolio/nlapack

[3] https://github.com/mateogianolio/vectorious/tree/opt-cblas

Re: Why I Still Use Python for High Performance Scientific Computing

#76

Post by harshnisar »

echlebek wrote: post #45
dagw wrote: post #39

Earlier quoted context omitted.

Nothing exotic. One of the changes for example was replacing a list of list with a set of tuples, which greatly sped up checking if an object was in the collection. Another change was using a generator comprehension and an included itertool function rather than hand rolled nested for loops.

Once you've exhausted all the low-hanging fruit, like people calling .keys() on dicts, or doing unnecessary linear searches, Cython really starts to shine. I've seen it perform ~40 times better than pure Python in time-consuming loops. We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.

Can you explain the context in which .keys() is called often and is not appropriate and the alternative?

Re: Why I Still Use Python for High Performance Scientific Computing

#77

Post by daemonk »

The interesting insight from the article is that python might be a good language for learning algorithms. The fast development time allows you to write a complete program (albeit clunky) without the pre-optimizing you might be tempted to do in other languages.

Re: Why I Still Use Python for High Performance Scientific Computing

#78

Post by cpbotha »

kfk wrote: post #69
cpbotha wrote: post #60

Earlier quoted context omitted.

I have first-hand experience with a BI-ish system, squarely targeted at the enterprise and doing quite well there, that we wrote using Django and a whole list of open source components. We did run into some resistance initially, because our stack is almost the opposite in every way of what our enterprise colleagues are used to. However, our development velocity, especially around analytical features and just in gener…

It would be interesting to hear about your experience, did you do a write up somewhere or could I email you with few questions? My project would be to put different data sources together + to allow users to upload their own structured data via Excel (think financial estimates). The current system has about 450 users, the next might have much more depending if it gets extended to other divisions.

You could send an email to the work address on my personal website (see HN profile). I can't promise that I'll be able to answer everything. :)

Re: Why I Still Use Python for High Performance Scientific Computing

#79

Post by dagw »

harshnisar wrote: post #76
echlebek wrote: post #45

Earlier quoted context omitted.

Once you've exhausted all the low-hanging fruit, like people calling .keys() on dicts, or doing unnecessary linear searches, Cython really starts to shine. I've seen it perform ~40 times better than pure Python in time-consuming loops. We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.

Can you explain the context in which .keys() is called often and is not appropriate and the alternative?

.keys() returns a list (in python2) so if you write

  for k in dict.keys():
    ...
then python first builds a list of all the keys, loops through them and then throws away the list. If the dict is large, this can be quite expensive. The correct way is to either use .iterkeys() which returns an iterator which generates the keys one at a time, or simply iterate directly over the dict, saving you need to first copy all the keys into a list you'll just throw away.

This has been 'fixed' in python3 and .keys() now returns an iterable view of the keys, and if you actually want a list of the keys you have to explicit and write list(dict.keys())

Re: Why I Still Use Python for High Performance Scientific Computing

#80

Post by superbatfish »

boulos wrote: post #18

If Numpy, Pandas, etc. were wrappable from JavaScript this could have easily been titled "Why I use Node.js for High Performance Scientific Computing". The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!". I agree some of the tooling and niceties are beyond a doubt best in breed…

You clearly do not understand software.
Post reply on HN