Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

41–50 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#41

Post by lqdc13 »

pbowyer wrote: post #38

> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast. What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io and https://tideways.io , it's felt a step backwards. I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree an…

cProfile takes a while to learn how to use well.

What didn't you like about line_profiler?

Here's a good guide on how to write fast(ish) code in Python: https://wiki.python.org/moin/PythonSpeed/PerformanceTips

Generally, the best strategy for me has been to use NumPy wherever possible and to avoid creating many complex objects. Best to use built in dicts or tuples for things that store data. Thus the only time I run into issues is when implementing algorithms in which case I usually isolate the slow function and turn it into a Cython module. Recently have been playing around with https://github.com/jboy/nim-pymod which seems like a much better solution.

Re: Why I Still Use Python for High Performance Scientific Computing

#43

Post by sukulaku »

wesm wrote: post #21
boulos wrote: post #18

If Numpy, Pandas, etc. were wrappable from JavaScript this could have easily been titled "Why I use Node.js for High Performance Scientific Computing". The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!". I agree some of the tooling and niceties are beyond a doubt best in breed…

Completely bizarre attitude (creator of pandas here).

Why is it bizarre?

His point is that the article shouldn't make it sound like "Python is fast", because the speed actually comes from the libraries that have been implemented in C.

Re: Why I Still Use Python for High Performance Scientific Computing

#44

Post by lmm »

wesm wrote: post #26
vog wrote: post #24

Earlier quoted context omitted.

Not disagreeing with you, but would you mind elaborating on that?

Claiming that using wrapped libraries written in another programming language (LAPACK, anyone?) is an inauthentic usage of the language (here, Python) is pedantic and unhelpful. C is just a wrapper for assembly, then, right?

Sometimes it's not - sometimes LAPACK is written in C. Sometimes your C is just a wrapper around FORTRAN and people should take this fact seriously: modern FORTRAN is often a better option than C for numerical workloads.

Re: Why I Still Use Python for High Performance Scientific Computing

#45

Post by echlebek »

dagw wrote: post #39
timClicks wrote: post #31

Earlier quoted context omitted.

Which data structures did you use? In Python, I tend to rely on dict, list, and set for 90% or more of my code. I wouldn't want to rely on structures written in pure Python.

Nothing exotic. One of the changes for example was replacing a list of list with a set of tuples, which greatly sped up checking if an object was in the collection. Another change was using a generator comprehension and an included itertool function rather than hand rolled nested for loops.

Once you've exhausted all the low-hanging fruit, like people calling .keys() on dicts, or doing unnecessary linear searches, Cython really starts to shine. I've seen it perform ~40 times better than pure Python in time-consuming loops.

We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.

Re: Why I Still Use Python for High Performance Scientific Computing

#46

Post by wesm »

sukulaku wrote: post #43
wesm wrote: post #21

Earlier quoted context omitted.

Completely bizarre attitude (creator of pandas here).

Why is it bizarre? His point is that the article shouldn't make it sound like "Python is fast", because the speed actually comes from the libraries that have been implemented in C.

By bizarre I mean impractical and unhelpful. What's the point of programming at all if we cannot leverage abstractions to make ourselves more productive?

I believe what the article says is that "Python has tools that enable a savvy user to achieve better results with less effort". Python is extremely popular in HPC settings (including supercomputers) for this reason. I see nothing disingenuous.

Re: Why I Still Use Python for High Performance Scientific Computing

#47

Post by wesm »

lmm wrote: post #44
wesm wrote: post #26

Earlier quoted context omitted.

Claiming that using wrapped libraries written in another programming language (LAPACK, anyone?) is an inauthentic usage of the language (here, Python) is pedantic and unhelpful. C is just a wrapper for assembly, then, right?

Sometimes it's not - sometimes LAPACK is written in C. Sometimes your C is just a wrapper around FORTRAN and people should take this fact seriously : modern FORTRAN is often a better option than C for numerical workloads.

I never stated what language LAPACK was written in.

Re: Why I Still Use Python for High Performance Scientific Computing

#48

Post by lmm »

But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of converting to/from numpy formats.

(And while I personally love Scala, there's nothing magic about it in this regard. There's no reason a language can't offer Python-like expressiveness and Java-like performance, and many modern languages do)

Re: Why I Still Use Python for High Performance Scientific Computing

#49

Post by lmm »

wesm wrote: post #47
lmm wrote: post #44

Earlier quoted context omitted.

Sometimes it's not - sometimes LAPACK is written in C. Sometimes your C is just a wrapper around FORTRAN and people should take this fact seriously : modern FORTRAN is often a better option than C for numerical workloads.

I never stated what language LAPACK was written in.

Then what was "C is just a wrapper for assembly, then, right?" supposed to mean?

Re: Why I Still Use Python for High Performance Scientific Computing

#50

Post by pjc50 »

olau wrote: post #37
vmarsy wrote: post #8

Earlier quoted context omitted.

All the points the author makes through the post are interesting, and Python is definitely great for protoyping, but I think the initial premise is false: "people don't tend to think of [Python] as a high performance language; for that you would want a compiled language -- ideally C or C++ but Java would do." Java is compiled to bytecode, but it isn't a "compiled language" since that bytecode has to be interpreted by…

"...and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code." True. But this assumes that time is not a constraint. I think you need to think of it this way (as a thought experiment): you start two programmers off, one in C and one in Python, both with a vague understanding of how to solve the problem and approximately the same skill level. Then after X ho…

Absolutely. For this kind of one-shot scientific computing, the only way the C programmer can win is if the Python programmer is sitting on their hands for weeks waiting for their program to run.
Post reply on HN