Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

141–150 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#141

Post by msellout »

lmm wrote: post #48

But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of c…

> But you can have both.

It seems you misread the article as saying Python created a trade-off. Perhaps the author shouldn't have tried to create a surprise ending. Regardless, the conclusion was that the Python implementation was both easier to code and more performant than the Java implementation.

Re: Why I Still Use Python for High Performance Scientific Computing

#142

Post by Fede_V »

vegabook wrote: post #59
pen2l wrote: post #22

This may be a liiiitle bit off-topic, but I really need to get it off my chest: Python for high-performance scientific computer works beautifully ... it's a dream. Scipy/numpy, matplotlib, pandas, ipython. They're all unbelievably awesome. It all just works. Except , when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible. So…

Interestingly though, Windows version of Numpy on Anaconda is quite a lot faster than what you'll get with Ubuntu, like 2x faster easily. I think it has to do with the fact that AVX instructions are used on the windows linalg routines that it binds to. If you buy MKL of course you get much more speed again.

Christian Gohlke distributes free NumPy binaries built against MKL.

Re: Why I Still Use Python for High Performance Scientific Computing

#143

Post by eonwe »

srean wrote: post #126
eonwe wrote: post #122

Earlier quoted context omitted.

For normal dense matrix laid out as a double[] and accessed directly as i* N_ROW + j probably won't get its loop check elided. For double[][] I would _think_ that happens more easily. But how are sparse matrices then generally laid out? A naive approach would be some hash map, perhaps with some locality, in which I don't see JIT problems.

There are some defacto standard formats such as CRS, CSC, list of tuples etc. Layout of the third should be obvious and it is not used much for cases where speed matters because one loses locality in this layout. For the other two they are laid out column after column (or row by row), row ids, and offsets to indicate the start and end of columns (rows).

Thanks, this helped. Using CSC and CRS is probably quite problematic with JVM bounds check elimination (or the lack of it). So if they would need to be used on JVM, I think it would be wise to drop the safety and use _sun.misc.Unsafe_ for unchecked array access.

Re: Why I Still Use Python for High Performance Scientific Computing

#144

Post by baldfat »

makeset wrote: post #129
baldfat wrote: post #94

Earlier quoted context omitted.

No squinting needed :) "R, at its heart, is a functional programming (FP) language. This means that it provides many tools for the creation and manipulation of functions. In particular, R has what’s known as first class functions. You can do anything with functions that you can do with vectors: you can assign them to variables, store them in lists, pass them as arguments to other functions, create them inside functio…

Yeah, lots of squinting needed, or a terribly shallow understanding of FP to say that. Merely providing first-class functions and map/reduce idioms does not a functional language make. Python has those, so is it also FP really? R is as much a FP language as Haskell or OCaml are imperative "at heart."

Sorry to STRONGLY disagree with you 100% but R clearly is a functional language and it might just be that you are unfamiliar with R or have not spent time with R.

Weak example is Wikipedia list of Functional Programming Languages list R as one. https://en.wikipedia.org/wiki/Functional_programming#R

R describes itself as a Functional programming Language:

The language syntax has a superficial similarity with C, but the semantics are of the FPL (functional programming language) variety with stronger affinities with Lisp and APL. In particular, it allows “computing on the language”, which in turn makes it possible to write functions that take expressions as input, something that is often useful for statistical modeling and graphics. https://cran.r-project.org/doc/manuals/r-release/R-lang.html

Academic Evidence: "The Journal Trends in Functional Programming" also list R as a functional programming language.

R is an environment and functional programming language for statistical data analysis and visualization. Largely unknown to the functional programming community, it is popular and influential in many empirical sciences.

http://link.springer.com/chapter/10.1007%2F978-3-642-40447-4...

The issue might be it isn't Haskel so it is not 100% functional but there are more Functional Programming Languages then Haskel.

PS This is why so many programs in R are poorly written because people try to force OO into R when they could do things Functional. That is also why my eyes bleed when I see people looping through everything in their R scripts.

Re: Why I Still Use Python for High Performance Scientific Computing

#145

Post by baldfat »

makeset wrote: post #129
baldfat wrote: post #94

Earlier quoted context omitted.

No squinting needed :) "R, at its heart, is a functional programming (FP) language. This means that it provides many tools for the creation and manipulation of functions. In particular, R has what’s known as first class functions. You can do anything with functions that you can do with vectors: you can assign them to variables, store them in lists, pass them as arguments to other functions, create them inside functio…

Yeah, lots of squinting needed, or a terribly shallow understanding of FP to say that. Merely providing first-class functions and map/reduce idioms does not a functional language make. Python has those, so is it also FP really? R is as much a FP language as Haskell or OCaml are imperative "at heart."

> "terribly shallow understanding of FP to say that"

That was very condescending and I am sure don't know who Hadley Wickham is? I wouldn't call his understanding "Shallow."

Re: Why I Still Use Python for High Performance Scientific Computing

#146

Post by KirinDave »

codygman wrote: post #137
KirinDave wrote: post #5

Earlier quoted context omitted.

The only functional langauges to gain much popularity here are OCaml and F#. Haskell tends not to be a go-to choice here because it... well... there are a lot of reasons. It's a tough language to learn compared to its competitors because it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful…

> it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful libraries used them. Err... can you qualify this? Based on my knowledge of Haskell this isn't a fair assessment at all.

Based on my knowledge, it's very fair. How many language extensions are available vs placed in common use? Where is the guidance on which to use? You WILL use at least 3 I can think of off hand very commonly.

And how many failed lens libraries preceeded the current (quite good) dominant library that still show up on google and even get included in? How many people still run versions of Yesod using now-unfavored abstractions?

The Haskell community really got its shit together over the last 3 years. Cabal got a lot better, a bunch of key libraries got good, our compiler is fixed some major bugs. But these developments do not magically erase people who have navigated the prior 5 years of Haskell where a lot of these technologies got established.

I love Haskell, I really do. I wish I could ship more code with Haskell, and like it as well. But the OP asked about why Haskell didn't take off. Its ecosystem was in what we might call a state of growth and flux at the time a lot of stream and batch computation systems were shipping.

Oh and uh, I think Haskell's community is full of some truly toxic people, an intersection of some of the ugliest and most arrogant personalities of the Scala world who's primary education strategy can only be called "negging." You will know these people when you find them. Compared to the Clojure community or the F# community or the Erlang community which are also forward thinking, interesting and skilled... it looks positively hostile. You may not care about that, but I refuse to endorse or use technologies who's community leadership is so toxic. Why, I've stopped using Linux in anything but legacy apps because I completely despair of that project from ever rising above is puerile and cliquish roots.

Re: Why I Still Use Python for High Performance Scientific Computing

#147

Post by IanCal »

corysama wrote: post #110
IanCal wrote: post #85

Earlier quoted context omitted.

> Numpy does 90% of the work Numpy and scipy have been the core of a huge amount of my optimisations. The first question I try and ask is "Could this be solved with matrix multiplications and summing?" Often the answer is "yes" and allows you to group a huge amount of calculations all together, and use the heavily optimised code available numpy/scipy. I recently swapped out something that was running at about 100 row…

http://www.vetta.org/2008/05/scipy-the-embarrassing-way-to-c... ;)

Fantastic!

Re: Why I Still Use Python for High Performance Scientific Computing

#148

Post by drpaulbrewer »

boulos wrote: post #18

If Numpy, Pandas, etc. were wrappable from JavaScript this could have easily been titled "Why I use Node.js for High Performance Scientific Computing". The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!". I agree some of the tooling and niceties are beyond a doubt best in breed…

Shameless plug...

If you want to import, export, tabulate, edit, plot and analyze CSV data in browser-side Javascript, here's something I wrote to paste numericJS, jqPlot, lzstring, and various html5 storage and file handling features together.

https://github.com/DrPaulBrewer/html5csv

Re: Why I Still Use Python for High Performance Scientific Computing

#149

Post by dafrankenstein2 »

kfk wrote: post #54

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking f…

maybe Play framework, which supports both Java and Scala, is also a good fit here!

Re: Why I Still Use Python for High Performance Scientific Computing

#150

Post by makeset »

baldfat wrote: post #144
makeset wrote: post #129

Earlier quoted context omitted.

Yeah, lots of squinting needed, or a terribly shallow understanding of FP to say that. Merely providing first-class functions and map/reduce idioms does not a functional language make. Python has those, so is it also FP really? R is as much a FP language as Haskell or OCaml are imperative "at heart."

Sorry to STRONGLY disagree with you 100% but R clearly is a functional language and it might just be that you are unfamiliar with R or have not spent time with R. Weak example is Wikipedia list of Functional Programming Languages list R as one. https://en.wikipedia.org/wiki/Functional_programming#R R describes itself as a Functional programming Language: The language syntax has a superficial similarity with C, but th…

Like many modern languages, R supports multiple programming paradigms, including sequential, object-oriented, and functional, but it should take more than a couple of functional features to declare something a full-blown FPL. A duck-billed platypus is not a duck.

Yes, you can write R code in a functional style in places when convenient, but one floor down it's still running that loop you didn't want to see. It's a language based on mutable values and sequential execution and looping, not pure function calls and recursion. The syntax itself is very much sequential, unlike stereotypical FPLs. Even the lexical scoping design relies fundamentally on mutable environments. So yes, people do seem to enjoy proudly declaring R a FPL citing a few features, but as I said, those features make R as much a FPL as Python.

Post reply on HN