I use Spark. Will using Python help a lot ?
Why I Still Use Python for High Performance Scientific Computing
131–140 of 158 posts
Re: Why I Still Use Python for High Performance Scientific Computing
#132But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of c…
Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI. I consider it an accident of history that Numpy, Scipy, Pandas, Scikits…
Correct me if I am wrong but Spark does not use JNI for Python. It uses py4j to allow Python programs to access Java Objects. All the core functionality is implemented in Scala/Java. py4j uses sockets to communicate. See https://www.py4j.org/about.html
Re: Why I Still Use Python for High Performance Scientific Computing
#133Earlier quoted context omitted.
First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter. Second, it is an inherent feature of the design of CPython that its C API allows tight integration with external libraries in C. Cython does not just glue C and Python together, it does this in a way which makes the integration easi…
> First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter. Sure, but that's how people construe the post, which I think the author knows too. The post could have been accurately titled "Python has certain libraries that are fast enough for HPSC" , but that wouldn't have generated nearly a…
Just because people construe the post that way doesn't mean they're right. Truth is not democratic.
You can make any title more accurate by making it longer and more specific, but picking a good title is a trade off against other things as well. A charitable reader would not necessarily insist on the narrow interpretation of Python the language you insist on.
And your title is not more accurate, because it's not just the existence of certain libraries, it's also the way they can be easily glued together and incrementally improved. Compare this to say Java, where you'd have to use JNI to use those same libraries, which is much more cumbersome to the point that most people prefer to write everything in Java.
> People want to see others say good things about their favourite language
To be honest I only think this only describes people that enjoy internet debates on programming languages. If what you are interested in is getting work done, you don't care about the merits of a particular tool in isolation but about end results. Hypothetically, you could find the "perfect" language but if it doesn't have users or libraries, it's of no use, and therefore the insistence of discussing only this or that language is pointless.
Re: Why I Still Use Python for High Performance Scientific Computing
#134Re: Why I Still Use Python for High Performance Scientific Computing
#135Earlier quoted context omitted.
Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI. I consider it an accident of history that Numpy, Scipy, Pandas, Scikits…
Correct me if I am wrong but Spark does not use JNI for Python. It uses py4j to allow Python programs to access Java Objects. All the core functionality is implemented in Scala/Java. py4j uses sockets to communicate. See https://www.py4j.org/about.html
That's my understanding as well. This is the reason why you are caught between the rock and a hard place on the Hadoop stack. Java code wont be efficient in terms of FLOPs and a popular and often effective escape hatch: code number crunching parts in C, C++ or Fortran, is also not very effective / convenient. Particularly so if it requires going back and forth over the bridge frequently because crossing the bridge has significant overhead. So typically you have to move majority of the core into the high performant language of choice and what remains is glue. If gluing is what I want to do, there are other languages that can give Java stiff competition. The core capability thing going for Java in this domain is HDFS, and its not that great a file system for big data.
Re: Why I Still Use Python for High Performance Scientific Computing
#136Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…
Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures, which is what you need to efficiently implement many numerical algorithms. But I'm a fan of Haskell's [REPA]( https://hackage.haskell.org/package/repa ) which provides multidimensional arrays with complete control over which operations are computed immediately and which are deferred and can be combined later with o…
> Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures
Can you provide some Haskell examples that are cumbersome, then provide an example that isn't in another language of your choice?
Also, have you tried using lens for this?
Re: Why I Still Use Python for High Performance Scientific Computing
#137Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…
The only functional langauges to gain much popularity here are OCaml and F#. Haskell tends not to be a go-to choice here because it... well... there are a lot of reasons. It's a tough language to learn compared to its competitors because it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful…
> it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful libraries used them.
Err... can you qualify this? Based on my knowledge of Haskell this isn't a fair assessment at all.
Re: Why I Still Use Python for High Performance Scientific Computing
#138Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…
The diff between Haskell and Python is not functional programming, I program functionally in Python, it is the type system. This is subjective, but having tried Elm with its Haskellish type system, it seems that the type system makes me think hard about things that ultimately don't matter much. It might depend on your problem, but I doubt it. Even doing numerical computation with Theano I find myself better off just…
> This is subjective, but having tried Elm with its Haskellish type system, it seems that the type system makes me think hard about things that ultimately don't matter much. It might depend on your problem, but I doubt it. Even doing numerical computation with Theano I find myself better off just executing subexpressions and seeing results and the shape of the tensors interactively is more useful than thinking really hard how the operations transform the shape.
If it is true that the type system, perhaps static type systems in general, make you think hard about things that ultimately don't matter much... why not go for the most dynamic and interactive language possible? Clojure perhaps.
I've been toying with the idea that either fully dynamic and as interactive as possible or full dependent typing is the generally right answer.
By right, I mean what you should reach for with most programming tasks if you aren't going to put effort into selecting one tailored to your task.
Re: Why I Still Use Python for High Performance Scientific Computing
#139Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere. From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable…
Because the Python ecosystem is huge, with real scientists writing real libraries to get stuff done. The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).
> The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).
There are "Real World" Haskell resources, in fact there is even a book named Real World Haskell[0], though I recommend Haskell Book[1] these days since it takes an approach of teaching from first principles.
Re: Why I Still Use Python for High Performance Scientific Computing
#140Earlier quoted context omitted.
"...and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code." True. But this assumes that time is not a constraint. I think you need to think of it this way (as a thought experiment): you start two programmers off, one in C and one in Python, both with a vague understanding of how to solve the problem and approximately the same skill level. Then after X ho…
I've tried to do a test like this and the results surprised me. While the testing conditions weren't perfect, I tracked how much time it took to port Fortran code to C++, and how much time it took to write the C++/CUDA optimized version. I would have expected writing all the optimized CUDA kernels would add significant time to the project, but in reality it was something like 1.3x for a solution 15x faster. I think a…
On the contrary, I pick Python to make the "testing, debugging, optimization, and design" faster.