Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

31–40 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#31

Post by timClicks »

dagw wrote: post #29
cygnus_a wrote: post #9

Earlier quoted context omitted.

Actually, even the subsection headings in bold give a very succinct summary: - Python has easy development ( https://xkcd.com/353/ ) - Great libraries (ie, free matlab) - Cython for efficiency via C - The algorithms themselves determine speediness (ie numerical methods)

The algorithms themselves determine speediness This is so important I wish people would focus more on it. I recently rewrote some Javascript code in (pure) python and got a good 2 orders of magnitude speed up on large inputs just by picking the right data structures and replacing an O(n^3) nested loop with an O(n log n) approach.

Which data structures did you use? In Python, I tend to rely on dict, list, and set for 90% or more of my code. I wouldn't want to rely on structures written in pure Python.

Re: Why I Still Use Python for High Performance Scientific Computing

#32

Post by pbowyer »

dagw wrote: post #29
cygnus_a wrote: post #9

Earlier quoted context omitted.

Actually, even the subsection headings in bold give a very succinct summary: - Python has easy development ( https://xkcd.com/353/ ) - Great libraries (ie, free matlab) - Cython for efficiency via C - The algorithms themselves determine speediness (ie numerical methods)

The algorithms themselves determine speediness This is so important I wish people would focus more on it. I recently rewrote some Javascript code in (pure) python and got a good 2 orders of magnitude speed up on large inputs just by picking the right data structures and replacing an O(n^3) nested loop with an O(n log n) approach.

Have you written a blog post about this? If not, care to? Trying to speed up a codebase, it's of interest to me (and no doubt others too).

Re: Why I Still Use Python for High Performance Scientific Computing

#33

Post by j-pb »

ntoshev wrote: post #20
cballard wrote: post #4

Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…

The diff between Haskell and Python is not functional programming, I program functionally in Python, it is the type system. This is subjective, but having tried Elm with its Haskellish type system, it seems that the type system makes me think hard about things that ultimately don't matter much. It might depend on your problem, but I doubt it. Even doing numerical computation with Theano I find myself better off just…

If you're using persistent datastructures like pyrsistent I could agree, but vanilla python is very unsuited for functional programming. Even JS has better lambdas.

Re: Why I Still Use Python for High Performance Scientific Computing

#34

Post by j-pb »

Tomte wrote: post #15
cballard wrote: post #4

Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…

Because the Python ecosystem is huge, with real scientists writing real libraries to get stuff done. The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).

C'mon zippers are not that hard, and really useful.

Lets say you want to do processing of some xml file. Normally you'd walk the tree and do manipulations in place. With zippers however, you can inspect every intermediary tree result, you can rethink you problem so that you walk the tree once to extract interesting information, then compute a changeset for the tree, maybe merge it with a differently computed changeset, and then apply the union of them.

I build a ocr system in clojure on zippers and it was a lot of fun. You could for example extract a list of all the words, with line wraps removed, then do the correction on that view/lense of your data, and reapply the changes without having to worry to reintroduce the pesky linewraps, because they were never removed from the original document.

Re: Why I Still Use Python for High Performance Scientific Computing

#35

Post by cdavid »

boulos wrote: post #18

If Numpy, Pandas, etc. were wrappable from JavaScript this could have easily been titled "Why I use Node.js for High Performance Scientific Computing". The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!". I agree some of the tooling and niceties are beyond a doubt best in breed…

Python in science is more than mere wrapping: it gives you new level(s) of abstractions, unrivalled in any other generic programming language AFAIK. If it were mere wrapping, other languages would have been able to catch up fairly quickly after all.

Big parts of numpy (and to a lesser degree pandas) are not in python, but lots of libraries on top of it are mostly python (scikit learn, scikits image, etc.). It is no more wrapping than lisp is wrapping the C VM.

Re: Why I Still Use Python for High Performance Scientific Computing

#36

Post by okpatil »

cygnus_a wrote: post #9
msellout wrote: post #2

Summary in the conclusion: "The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]

Actually, even the subsection headings in bold give a very succinct summary: - Python has easy development ( https://xkcd.com/353/ ) - Great libraries (ie, free matlab) - Cython for efficiency via C - The algorithms themselves determine speediness (ie numerical methods)

Just typed "import antigravity" in python repl. It took me to the same link. :-)

Re: Why I Still Use Python for High Performance Scientific Computing

#37

Post by olau »

vmarsy wrote: post #8
msellout wrote: post #2

Summary in the conclusion: "The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]

All the points the author makes through the post are interesting, and Python is definitely great for protoyping, but I think the initial premise is false: "people don't tend to think of [Python] as a high performance language; for that you would want a compiled language -- ideally C or C++ but Java would do." Java is compiled to bytecode, but it isn't a "compiled language" since that bytecode has to be interpreted by…

"...and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code."

True. But this assumes that time is not a constraint.

I think you need to think of it this way (as a thought experiment): you start two programmers off, one in C and one in Python, both with a vague understanding of how to solve the problem and approximately the same skill level. Then after X hours, you stop both and see how far they've gotten.

I think the argument then is that you might find that the C programmer hasn't yet solved the problem, whereas the Python programmer might have a solution + a deeper understanding of where the slow parts are, and started optimizing those.

So in the end the C programmer may win, but perhaps it's not about winning in this sense but more about how fast you can get to a point where you can move on to the next thing.

Re: Why I Still Use Python for High Performance Scientific Computing

#38

Post by pbowyer »

> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast.

What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io and https://tideways.io, it's felt a step backwards.

I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree and KCachegrind. I've found the cProfile output confusing when it crosses the Python-C barrier for numpy, sklearn etc.

Re: Why I Still Use Python for High Performance Scientific Computing

#39

Post by dagw »

timClicks wrote: post #31
dagw wrote: post #29

Earlier quoted context omitted.

The algorithms themselves determine speediness This is so important I wish people would focus more on it. I recently rewrote some Javascript code in (pure) python and got a good 2 orders of magnitude speed up on large inputs just by picking the right data structures and replacing an O(n^3) nested loop with an O(n log n) approach.

Which data structures did you use? In Python, I tend to rely on dict, list, and set for 90% or more of my code. I wouldn't want to rely on structures written in pure Python.

Nothing exotic. One of the changes for example was replacing a list of list with a set of tuples, which greatly sped up checking if an object was in the collection. Another change was using a generator comprehension and an included itertool function rather than hand rolled nested for loops.

Re: Why I Still Use Python for High Performance Scientific Computing

#40

Post by echlebek »

pbowyer wrote: post #38

> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast. What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io and https://tideways.io , it's felt a step backwards. I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree an…

I've optimized quite a bit of code with Cython. I do something like:

python -m cProfile module.py | gprof2dot -f pstats | python -m xdot -

Post reply on HN