Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

91–100 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#91

Post by restalis »

fsloth wrote: post #13
tokipin wrote: post #12

Earlier quoted context omitted.

No need to kid. Self-described "FP" languages are a pain in the ass for the type of programming scientists do. I don't even know what kind of programming those languages aren't a pain in the ass for.

"I don't even know what kind of programming those languages aren't a pain in the ass for." If we limit the discussion to ML languages, they are generally very nice e.g. for datastructure transforms (including interpreters and compilers). Static typing with type inference makes implementations short, and, if written in good style, are correct when they compile. There are various other reasons why one would not want to…

"I would say learning self-described fp-languages have made me a better programmer who values static typing more than before."

FP paradigm has indeed didactic value and the experience gained in a functional language may translate into a plus of productivity ...in other languages. When you have to get things done, the functional language's embedded arm-twisting (well, mind-twisting actually) doesn't help! That's when you'll either abandon the (idea of) "pure" functional language for a multi-paradigm language (that will allow you to keep a mainly functional style with limited deviations when the situation warrants it) or you'll keep pushing yourself and throw the blame on you for not being good enough with a language that wasn't designed from very beginning for much anything but the FP standpoint (for the sake of it).

Re: Why I Still Use Python for High Performance Scientific Computing

#92

Post by sergiosgc »

dagw wrote: post #79
harshnisar wrote: post #76

Earlier quoted context omitted.

Can you explain the context in which .keys() is called often and is not appropriate and the alternative?

.keys() returns a list (in python2) so if you write for k in dict.keys(): ... then python first builds a list of all the keys, loops through them and then throws away the list. If the dict is large, this can be quite expensive. The correct way is to either use .iterkeys() which returns an iterator which generates the keys one at a time, or simply iterate directly over the dict, saving you need to first copy all the k…

Just to clarify your python 2 note: In python 3 keys() returns an iterator, so there is no penalty (i.e. iterkeys was dropped, keys assumed iterkeys interface).

The equivalent python2 behaviour can be obtained using list(somedict.keys())

Re: Why I Still Use Python for High Performance Scientific Computing

#93

Post by kitd »

iamsohungry wrote: post #83
kitd wrote: post #73

Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have very different requirements so a combination of a tool that can do all the data handling easily (ie Python) + a tool that can throw the CPU at a numerical problem (ie C) will work as a great combination. In contrast, if you work in Java, you are trying to use…

> My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node? Absolutely not. Python has a much more mature set of libraries which are much better designed, better tooling, and a much more reasonable type system, and the community has only recently begun to be polluted by Web 2.0 "move fas…

Yes you're right, Node wouldn't be good.

Then I remembered Perl and realised that data pre-/post-processing is precisely why it was invented in the first place.

Perl + Rust would be an interesting combo IMHO.

Re: Why I Still Use Python for High Performance Scientific Computing

#94

Post by baldfat »

mcepl wrote: post #57
optimiz3 wrote: post #6

Earlier quoted context omitted.

Probably because functional languages don't actually at excel scientific computing relative to Python, C#, Java, and JavaScript. :D Edit: I kid, but most of the languages I just mentioned have very fast native compilers, easy ways of invoking low-level interop, functional-style libraries if you want them, and (except for Python) C-like syntax making it easy to cut-and-paste.

Actually, if you squint a lot, R (or S) is a functional language. And it shows, crazily powerful for something, absolutely horrible for something else.

No squinting needed :)

"R, at its heart, is a functional programming (FP) language. This means that it provides many tools for the creation and manipulation of functions. In particular, R has what’s known as first class functions. You can do anything with functions that you can do with vectors: you can assign them to variables, store them in lists, pass them as arguments to other functions, create them inside functions, and even return them as the result of a function."

http://adv-r.had.co.nz/Functional-programming.html

Re: Why I Still Use Python for High Performance Scientific Computing

#95

Post by eonwe »

srean wrote: post #51
lmm wrote: post #48

But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of c…

Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI. I consider it an accident of history that Numpy, Scipy, Pandas, Scikits…

Could you elaborate a bit on how JVM hinders reaching maximum FLOPS when multiplying sparse matrices?

I could think some examples where lacking SSE/AVX support would hinder it, but I don't see the connection with sparse matrices.

Re: Why I Still Use Python for High Performance Scientific Computing

#96

Post by ajuc »

sukulaku wrote: post #84
andreasvc wrote: post #81

Earlier quoted context omitted.

First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter. Second, it is an inherent feature of the design of CPython that its C API allows tight integration with external libraries in C. Cython does not just glue C and Python together, it does this in a way which makes the integration easi…

> First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter. Sure, but that's how people construe the post, which I think the author knows too. The post could have been accurately titled "Python has certain libraries that are fast enough for HPSC" , but that wouldn't have generated nearly a…

Obivously clojure, being written in Java (at least the intersting parts - collections and multithreading) - isn't fast (if Python isn't fast).

Re: Why I Still Use Python for High Performance Scientific Computing

#97

Post by spot »

kitd wrote: post #73

Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have very different requirements so a combination of a tool that can do all the data handling easily (ie Python) + a tool that can throw the CPU at a numerical problem (ie C) will work as a great combination. In contrast, if you work in Java, you are trying to use…

Agreed. It is often true that different parts of a problem are best solved in different languages. That's one reason we built Beaker and released it as open source: http://BeakerNotebook.com. Its polyglot architecture and autotranslation makes working with multiple languages easy.

Re: Why I Still Use Python for High Performance Scientific Computing

#98

Post by sukulaku »

ajuc wrote: post #96
sukulaku wrote: post #84

Earlier quoted context omitted.

> First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter. Sure, but that's how people construe the post, which I think the author knows too. The post could have been accurately titled "Python has certain libraries that are fast enough for HPSC" , but that wouldn't have generated nearly a…

Obivously clojure, being written in Java (at least the intersting parts - collections and multithreading) - isn't fast (if Python isn't fast).

Right, but I wouldn't claim it is. I'd say something like "The JVM is fast" :)

Re: Why I Still Use Python for High Performance Scientific Computing

#99

Post by mcnamaratw »

sukulaku wrote: post #43
wesm wrote: post #21

Earlier quoted context omitted.

Completely bizarre attitude (creator of pandas here).

Why is it bizarre? His point is that the article shouldn't make it sound like "Python is fast", because the speed actually comes from the libraries that have been implemented in C.

To be fair, the article doesn't say "Python is fast." It says "Python is slow," and then goes on to say something like "But if most of your heavy-duty calculation involves calling prewritten C routines, you can do your development work in Python without getting killed on performance."

Language discussions always seem to go down the same dismal tube in the end: "Is C > Python, or is Python > C?"

Re: Why I Still Use Python for High Performance Scientific Computing

#100

Post by IanCal »

IanCal wrote: post #85
echlebek wrote: post #45

Earlier quoted context omitted.

Once you've exhausted all the low-hanging fruit, like people calling .keys() on dicts, or doing unnecessary linear searches, Cython really starts to shine. I've seen it perform ~40 times better than pure Python in time-consuming loops. We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.

> Numpy does 90% of the work Numpy and scipy have been the core of a huge amount of my optimisations. The first question I try and ask is "Could this be solved with matrix multiplications and summing?" Often the answer is "yes" and allows you to group a huge amount of calculations all together, and use the heavily optimised code available numpy/scipy. I recently swapped out something that was running at about 100 row…

In fairness, I should point out that the really slow version was also written by me :)
Post reply on HN