Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

51–60 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#51

Post by srean »

lmm wrote: post #48

But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of c…

Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI.

I consider it an accident of history that Numpy, Scipy, Pandas, Scikits got written for Python and not Lua. Thanks to Luajit I think Lua would have been a better choice. Now that cause has been taken up by Torch and Julia.

JVM by itself is terrible for reaching close to the FLOPS that the CPU is capable of. Try sparse matrix multiply with it and see it for yourself.

Re: Why I Still Use Python for High Performance Scientific Computing

#52

Post by yomritoyj »

cballard wrote: post #4

Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…

Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures, which is what you need to efficiently implement many numerical algorithms. But I'm a fan of Haskell's [REPA](https://hackage.haskell.org/package/repa) which provides multidimensional arrays with complete control over which operations are computed immediately and which are deferred and can be combined later with other operations. This allows many computations to be expressed in a high-level array-oriented way without incurring the cost of creating huge temporaries. C++ template libraries like Eigen can do this, but give much less control to the user on what is deferred.

Re: Why I Still Use Python for High Performance Scientific Computing

#53

Post by syllogism »

pbowyer wrote: post #38

> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast. What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io and https://tideways.io , it's felt a step backwards. I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree an…

After using the profiler to look at the right piece of code, an important part of the process when optimising with Cython is:

cython -a module.pyx && open module.html

the -a command produces an annotated html file, which lets you see how Cython is compiling your Python into C. There's a colour coding to give you an indication of which bits are fully typed and being translated directly, and which parts are hitting the CPython API at run-time (and hence will be slow).

Re: Why I Still Use Python for High Performance Scientific Computing

#54

Post by kfk »

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking from a company perspective, the drawback python has is, strangely, the lack of paid for alternatives. It's not that people in companies don't trust open source (hadoop is becoming big here too), but one wonders if the developers will be able to find the support they need in case any issue arise from a free library.

Re: Why I Still Use Python for High Performance Scientific Computing

#55

Post by [deleted] »

dagw wrote: post #29
cygnus_a wrote: post #9

Earlier quoted context omitted.

Actually, even the subsection headings in bold give a very succinct summary: - Python has easy development ( https://xkcd.com/353/ ) - Great libraries (ie, free matlab) - Cython for efficiency via C - The algorithms themselves determine speediness (ie numerical methods)

The algorithms themselves determine speediness This is so important I wish people would focus more on it. I recently rewrote some Javascript code in (pure) python and got a good 2 orders of magnitude speed up on large inputs just by picking the right data structures and replacing an O(n^3) nested loop with an O(n log n) approach.

[deleted]

Re: Why I Still Use Python for High Performance Scientific Computing

#56

Post by Minick »

kfk wrote: post #54

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking f…

Python is fine but it can be more work...

Re: Why I Still Use Python for High Performance Scientific Computing

#57

Post by mcepl »

optimiz3 wrote: post #6
cballard wrote: post #4

Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…

Probably because functional languages don't actually at excel scientific computing relative to Python, C#, Java, and JavaScript. :D Edit: I kid, but most of the languages I just mentioned have very fast native compilers, easy ways of invoking low-level interop, functional-style libraries if you want them, and (except for Python) C-like syntax making it easy to cut-and-paste.

Actually, if you squint a lot, R (or S) is a functional language. And it shows, crazily powerful for something, absolutely horrible for something else.

Re: Why I Still Use Python for High Performance Scientific Computing

#58

Post by mcepl »

pen2l wrote: post #22

This may be a liiiitle bit off-topic, but I really need to get it off my chest: Python for high-performance scientific computer works beautifully ... it's a dream. Scipy/numpy, matplotlib, pandas, ipython. They're all unbelievably awesome. It all just works. Except , when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible. So…

I guess, if you don't know what to do with your money, you can get the similar results on any real operating system (e.g., Mac OS X).

Re: Why I Still Use Python for High Performance Scientific Computing

#59

Post by vegabook »

pen2l wrote: post #22

This may be a liiiitle bit off-topic, but I really need to get it off my chest: Python for high-performance scientific computer works beautifully ... it's a dream. Scipy/numpy, matplotlib, pandas, ipython. They're all unbelievably awesome. It all just works. Except , when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible. So…

Interestingly though, Windows version of Numpy on Anaconda is quite a lot faster than what you'll get with Ubuntu, like 2x faster easily. I think it has to do with the fact that AVX instructions are used on the windows linalg routines that it binds to. If you buy MKL of course you get much more speed again.

Re: Why I Still Use Python for High Performance Scientific Computing

#60

Post by cpbotha »

kfk wrote: post #54

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking f…

I have first-hand experience with a BI-ish system, squarely targeted at the enterprise and doing quite well there, that we wrote using Django and a whole list of open source components.

We did run into some resistance initially, because our stack is almost the opposite in every way of what our enterprise colleagues are used to. However, our development velocity, especially around analytical features and just in general, has made a most gratifying impact.

(I have to add that the 5-man dev team we have working on this is stellar. It's hard to determine scientifically what the interaction is between team quality and the choice of a Python-oriented software stack. See Paul Graham's essays for more discussion on that point.)

In terms of support: There are many highly professional often boutique software agencies that can support Django systems, if you're not around. To my mind this is even better than the normal commercial support you get from a different vendor for each different component in your commercial enterprise system.

Post reply on HN