Absolutely and it is even easier if you use Ceemple for your IDE
Why I Still Use Python for High Performance Scientific Computing
111–120 of 158 posts
Re: Why I Still Use Python for High Performance Scientific Computing
#112Earlier quoted context omitted.
Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI. I consider it an accident of history that Numpy, Scipy, Pandas, Scikits…
Could you elaborate a bit on how JVM hinders reaching maximum FLOPS when multiplying sparse matrices? I could think some examples where lacking SSE/AVX support would hinder it, but I don't see the connection with sparse matrices.
What make sparse matrices harder to JIT is proving that the loop bounds will not be exceeded, so all accesses re bound checked. In any case in my experience array bounds checking and escape analysis never gave the boost that theory and JVM fans promise. So even normal matrix multiply will trail behind. That said Hotspot JVM is possibly one of the most optimized VMs we have got.
A structural problem of JVM is that its runtime semantics is over-specified, there is very little room for the JIT to do its stuff. For example function arguments are evaluated right to left, there goes an opportunity for parallelism.
Re: Why I Still Use Python for High Performance Scientific Computing
#113Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…
> Why isn't Haskell, or any other functional language, popular for this sort of thing?
Because some of the scientists doing this are bellow average programmers. Look at the author singing praises to Python while benefiting from C, Fortran and Cython. He doesn't understand that what actually matters is what's under the hood, and you want him to learn functional programming and rewrite his algorithms?
Re: Why I Still Use Python for High Performance Scientific Computing
#114A very beginner Java programmer here. It's a nicely organized notebook, great demo, but: seems like a lot of effort was put into optimizing the python efforts, and none for Java. Isn't that an unfair comparison? My real question is, is it so much easier to do this excercise in Python than Java, assuming equal proficiency in either case?
> it became necessary to see how our python version compared to the high performance reference implementation in Java.
It sounds like the Java version was also optimized - though it's hard to say to what degree. I think in general if you're doing a lot of numerical computing and matrix operations, the basic java builtins are not going to cut it anyway, and you're going to end up using something like ND4j or JBLAS to get comparable performance to something like numpy, in which case I have hard time imagining it could possibly be /easier/ than numpy.
Granted, the cython stuff is more "advanced", and you probably would struggle a bit (or just would not care for the task) if you were a researcher with a surface understanding of what's really going on, but it's no big deal for a programmer. The other thing is that the defaults usually work well enough that you don't really have to pull out the big guns most of the time. You can also decide to do your optimization progressively. "Let me rewrite this part in cython" is a much quicker win than "let me rewrite all of this to run on top of platform X" or something.
Also it definitely does depend on the kind of problem. For example, the JVM does have awesome tools for distributed computing and streams like http://akka.io/, vs python is more lacking.
In conclusion though, I've been always very pleasantly surprised by how far I can get away with by just dumping larger and larger datasets (current record is in the hundreds of GB for me) into the python / pandas / scipy / numpy stack, and how much of a pleasure it is to use compared to anything else. To toss that away, I'd want to have tried and failed at a problem with the python stack first.
Re: Why I Still Use Python for High Performance Scientific Computing
#115Earlier quoted context omitted.
Absolutely. For this kind of one-shot scientific computing, the only way the C programmer can win is if the Python programmer is sitting on their hands for weeks waiting for their program to run.
Unless (1) the relevant library happens to be written in C and there are no python bindings and the problem is simple OR (2) there is an existing solution which is 95% complete in C and one needs to write from scratch in python. I've never come across situation (1) with a new project. Situation (2) is quite common.
But in those cases, the Python programmer could just wrap the C library in Python, which I suspect is often a lot easier than starting from scratch.
Re: Why I Still Use Python for High Performance Scientific Computing
#116Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…
> Why isn't Haskell, or any other functional language, popular for this sort of thing? Because some of the scientists doing this are bellow average programmers. Look at the author singing praises to Python while benefiting from C, Fortran and Cython. He doesn't understand that what actually matters is what's under the hood, and you want him to learn functional programming and rewrite his algorithms?
That seems very unfair. I have no reason to think the author doesn't understand that Python libraries are built on carefully written C and Fortran, and based on the article they seem to be a perfectly competent programmer.
Re: Why I Still Use Python for High Performance Scientific Computing
#117Couple of questions:
- Could this be/Was this developed in Python 3.x
- what is this "notebook" he keeps on referring to?
Re: Why I Still Use Python for High Performance Scientific Computing
#118This may be a liiiitle bit off-topic, but I really need to get it off my chest: Python for high-performance scientific computer works beautifully ... it's a dream. Scipy/numpy, matplotlib, pandas, ipython. They're all unbelievably awesome. It all just works. Except , when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible. So…
This is a problem for many languages that have external libs requiring local compilation (C, C++, etc.), given many of them are principally developed on Linux, with Windows compatibility being an afterthought. Its a problem not unique to Python.
Re: Why I Still Use Python for High Performance Scientific Computing
#119Couple of questions: - Could this be/Was this developed in Python 3.x - what is this "notebook" he keeps on referring to?
The "notebook" he refers to is the web page itself. It's a format that's becoming popular among Python / data science folks: a web-based, often interactive layout with code and results embedded in it. See "Jupyter Notebook", the viewer for which is what you're viewing as you read this.
Re: Why I Still Use Python for High Performance Scientific Computing
#120Earlier quoted context omitted.
All the points the author makes through the post are interesting, and Python is definitely great for protoyping, but I think the initial premise is false: "people don't tend to think of [Python] as a high performance language; for that you would want a compiled language -- ideally C or C++ but Java would do." Java is compiled to bytecode, but it isn't a "compiled language" since that bytecode has to be interpreted by…
The major implementations of the JVM are jited, not interpreted. Yes, you get great performance with jitting, tho not necessarily without unpredictable GC pauses and without a loss of energy efficiency from having to run a jit alongside your actual code.
GC pauses have little to nothing to do JIT process itself. You can get JVMs w/ read barriers and soft-realtime guarantees.[0][1]
Please note, gc and jit are quite a different thing, even if the gc needs help from the jit.
[0]: https://www-01.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/... [1]: https://www.azul.com/products/zing/