Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

101–110 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#101

Post by hughw »

tokipin wrote: post #12
optimiz3 wrote: post #6

Earlier quoted context omitted.

Probably because functional languages don't actually at excel scientific computing relative to Python, C#, Java, and JavaScript. :D Edit: I kid, but most of the languages I just mentioned have very fast native compilers, easy ways of invoking low-level interop, functional-style libraries if you want them, and (except for Python) C-like syntax making it easy to cut-and-paste.

No need to kid. Self-described "FP" languages are a pain in the ass for the type of programming scientists do. I don't even know what kind of programming those languages aren't a pain in the ass for.

Using Scala, I've found the ability to pass around first class functions, and the default immutability of variables, enables me to make profound algorithmic optimizations I wouldn't undertake in C++ et al. Then, if the computation generates too much garbage or goes slowly in a loop, I can then take shortcuts, like using mutable local variables, to make that section go faster.

Re: Why I Still Use Python for High Performance Scientific Computing

#102

Post by grndn »

kfk wrote: post #54

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking f…

How about F#? It feels like Python and has a Pandas equivalent in Deedle (https://bluemountaincapital.github.io/Deedle/), but fits well in the .NET ecosystem, including using SqlServer effortlessly. This report is a good starting point: http://fslab.org/report/

Re: Why I Still Use Python for High Performance Scientific Computing

#103

Post by mr_overalls »

kitd wrote: post #73

Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have very different requirements so a combination of a tool that can do all the data handling easily (ie Python) + a tool that can throw the CPU at a numerical problem (ie C) will work as a great combination. In contrast, if you work in Java, you are trying to use…

Julia could be a contender once a few bugs are worked out.

Re: Why I Still Use Python for High Performance Scientific Computing

#104

Post by banku_brougham »

A very beginner Java programmer here. It's a nicely organized notebook, great demo, but: seems like a lot of effort was put into optimizing the python efforts, and none for Java. Isn't that an unfair comparison?

My real question is, is it so much easier to do this excercise in Python than Java, assuming equal proficiency in either case?

Re: Why I Still Use Python for High Performance Scientific Computing

#105

Post by RogerL »

pen2l wrote: post #82
dagw wrote: post #64

Earlier quoted context omitted.

FUD. I've been using Python on windows for years and between Anaconda and Christoph Gohlke's python packages and I've yet to run into something that didn't just work.

Yeah with Anaconda it's not as bad. But no-one told me that's what I needed if I wanted all the science packages to work on Windows... I'd never even heard of Anaconda before this. It took all of my blood and tears for weeks before I got everything fixed. So I guess what's wrong here is the lack of documentation.

NumPy/SciPy's install page tells you to use Anaconda (or equivalent) for Windows, and build from repositories for Linux.

http://www.scipy.org/install.html

Granted, it does not say "and by the way, when you do that you get tons of other great things like pandas, matplotlib, etc". But they are very clear about not trying to do this yourself.

Re: Why I Still Use Python for High Performance Scientific Computing

#106

Post by AndrewGarner »

kfk wrote: post #54

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking f…

I would like to second the suggestion for F#. It has a nice balance in that it straddles the line between being an OSS ecosystem and an enterprise ecosystem with paid libraries. FsLab and Deedle got mentioned already but the F# website has a lot of great resources too: http://fsharp.org/guides/data-science/index.html

Re: Why I Still Use Python for High Performance Scientific Computing

#107

Post by RogerL »

sukulaku wrote: post #67
wesm wrote: post #46

Earlier quoted context omitted.

By bizarre I mean impractical and unhelpful. What's the point of programming at all if we cannot leverage abstractions to make ourselves more productive? I believe what the article says is that "Python has tools that enable a savvy user to achieve better results with less effort". Python is extremely popular in HPC settings (including supercomputers) for this reason. I see nothing disingenuous.

Well, the word "bizarre" has a commonly understood meaning, but you somehow decided to use it to mean something completely different. That's a bit bizarre :P But the article is titled: "Why I Still Use Python for High Performance Scientific Computing", and it gives the impression that Python - the language - is fast enough for HPSC. In reality, the reason why he "still" uses Python is that the libraries are fast enou…

The question is: what language should I use for HPC.

An answer is Python.

If you use C for HPC (the way you mean it) it will be slow as a dog. Because HPC is done with BLAS, LAPACK, etc. If you do HPC in C you are calling into BLAS and LAPACK. They are Fortran libraries. Anything else is dog slow in comparison (a slight stretch, but not by much).

So, the "real" answer is Fortran. But not a lot of people are sitting at their desk writing Fortran for scientific computing these days. They are writing Python, C, or C++ for the most part. All of which use libraries that call into the Fortran libraries.

That's all this article is saying, and we all understand it. No one is wringing their hands about what language the libraries are written in unless you are trying to write one of those libraries.

Working professionals use Python for scientific software because it excels at data munging, making reports, and very high speed computation. Everyone gets that you need to add some libraries beyond the libraries provided from the base install.

Re: Why I Still Use Python for High Performance Scientific Computing

#108

Post by netheril96 »

banku_brougham wrote: post #104

A very beginner Java programmer here. It's a nicely organized notebook, great demo, but: seems like a lot of effort was put into optimizing the python efforts, and none for Java. Isn't that an unfair comparison? My real question is, is it so much easier to do this excercise in Python than Java, assuming equal proficiency in either case?

Java has no operator overloading. Many Java developers vehemently oppose addition of operator overloading into the language, as if it were the root of all evil. The lack of that feature results in convoluted function calls where a clear and concise math expression would suffice. Consequently, not many people choose Java for math tasks, and not many people write math libraries for Java.

That is just my guess.

Re: Why I Still Use Python for High Performance Scientific Computing

#109

Post by [deleted] »

j-pb wrote: post #34
Tomte wrote: post #15

Earlier quoted context omitted.

Because the Python ecosystem is huge, with real scientists writing real libraries to get stuff done. The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).

C'mon zippers are not that hard, and really useful. Lets say you want to do processing of some xml file. Normally you'd walk the tree and do manipulations in place. With zippers however, you can inspect every intermediary tree result, you can rethink you problem so that you walk the tree once to extract interesting information, then compute a changeset for the tree, maybe merge it with a differently computed changese…

[deleted]

Re: Why I Still Use Python for High Performance Scientific Computing

#110

Post by corysama »

IanCal wrote: post #85
echlebek wrote: post #45

Earlier quoted context omitted.

Once you've exhausted all the low-hanging fruit, like people calling .keys() on dicts, or doing unnecessary linear searches, Cython really starts to shine. I've seen it perform ~40 times better than pure Python in time-consuming loops. We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.

> Numpy does 90% of the work Numpy and scipy have been the core of a huge amount of my optimisations. The first question I try and ask is "Could this be solved with matrix multiplications and summing?" Often the answer is "yes" and allows you to group a huge amount of calculations all together, and use the heavily optimised code available numpy/scipy. I recently swapped out something that was running at about 100 row…

http://www.vetta.org/2008/05/scipy-the-embarrassing-way-to-c... ;)
Post reply on HN