Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

81–90 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#81

Post by andreasvc »

sukulaku wrote: post #67
wesm wrote: post #46

Earlier quoted context omitted.

By bizarre I mean impractical and unhelpful. What's the point of programming at all if we cannot leverage abstractions to make ourselves more productive? I believe what the article says is that "Python has tools that enable a savvy user to achieve better results with less effort". Python is extremely popular in HPC settings (including supercomputers) for this reason. I see nothing disingenuous.

Well, the word "bizarre" has a commonly understood meaning, but you somehow decided to use it to mean something completely different. That's a bit bizarre :P But the article is titled: "Why I Still Use Python for High Performance Scientific Computing", and it gives the impression that Python - the language - is fast enough for HPSC. In reality, the reason why he "still" uses Python is that the libraries are fast enou…

First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter.

Second, it is an inherent feature of the design of CPython that its C API allows tight integration with external libraries in C. Cython does not just glue C and Python together, it does this in a way which makes the integration easier and safer than doing it by hand (e.g., generating correct reference counting code). All of this is a direct benefit of Python and its scientific ecosystem.

Re: Why I Still Use Python for High Performance Scientific Computing

#82

Post by pen2l »

dagw wrote: post #64
pen2l wrote: post #22

This may be a liiiitle bit off-topic, but I really need to get it off my chest: Python for high-performance scientific computer works beautifully ... it's a dream. Scipy/numpy, matplotlib, pandas, ipython. They're all unbelievably awesome. It all just works. Except , when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible. So…

FUD. I've been using Python on windows for years and between Anaconda and Christoph Gohlke's python packages and I've yet to run into something that didn't just work.

Yeah with Anaconda it's not as bad. But no-one told me that's what I needed if I wanted all the science packages to work on Windows... I'd never even heard of Anaconda before this. It took all of my blood and tears for weeks before I got everything fixed. So I guess what's wrong here is the lack of documentation.

Re: Why I Still Use Python for High Performance Scientific Computing

#83

Post by iamsohungry »

kitd wrote: post #73

Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have very different requirements so a combination of a tool that can do all the data handling easily (ie Python) + a tool that can throw the CPU at a numerical problem (ie C) will work as a great combination. In contrast, if you work in Java, you are trying to use…

> My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node?

Absolutely not. Python has a much more mature set of libraries which are much better designed, better tooling, and a much more reasonable type system, and the community has only recently begun to be polluted by Web 2.0 "move fast and break things" mentality. The Node ecosystem is built on that mentality, and while outlier developers at the front line of the JS community are able to be extraordinarily productive in JS/Node, developers building user-facing programs just get bogged down in breaking changes in dependencies, buggy and poorly-designed 0.x libraries, untraceable framework code that prioritizes configuration over convention, and intractable errors caused by a broken type system (this final issue is significantly improved in ES6).

Source: Worked in Python 2.x for a few years, worked in Node for a few years, currently work in Python 3.x and Node (general design is to limit Node to compiling and testing the browser-based part of our product). JS/Node is maybe 20% of our code, 20% of our features, 50% of our dev time, and 80% of our bugs.

Re: Why I Still Use Python for High Performance Scientific Computing

#84

Post by sukulaku »

andreasvc wrote: post #81
sukulaku wrote: post #67

Earlier quoted context omitted.

Well, the word "bizarre" has a commonly understood meaning, but you somehow decided to use it to mean something completely different. That's a bit bizarre :P But the article is titled: "Why I Still Use Python for High Performance Scientific Computing", and it gives the impression that Python - the language - is fast enough for HPSC. In reality, the reason why he "still" uses Python is that the libraries are fast enou…

First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter. Second, it is an inherent feature of the design of CPython that its C API allows tight integration with external libraries in C. Cython does not just glue C and Python together, it does this in a way which makes the integration easi…

> First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter.

Sure, but that's how people construe the post, which I think the author knows too.

The post could have been accurately titled "Python has certain libraries that are fast enough for HPSC", but that wouldn't have generated nearly as much interest as "Why I still use Python for HPSC".

People want to see others say good things about their favourite language, which in this case is Python.

I knew where the article was going (with the libraries), but still wanted to read it because even I wanted to see someone compliment Python itself, because it used to be my "primary language".

Python is a good language, but to the extent you're not using those highly optimized "C-libraries", it just doesn't perform well, and it's not suited for concurrency. That's a part of why I switched to Clojure.

Re: Why I Still Use Python for High Performance Scientific Computing

#85

Post by IanCal »

echlebek wrote: post #45
dagw wrote: post #39

Earlier quoted context omitted.

Nothing exotic. One of the changes for example was replacing a list of list with a set of tuples, which greatly sped up checking if an object was in the collection. Another change was using a generator comprehension and an included itertool function rather than hand rolled nested for loops.

Once you've exhausted all the low-hanging fruit, like people calling .keys() on dicts, or doing unnecessary linear searches, Cython really starts to shine. I've seen it perform ~40 times better than pure Python in time-consuming loops. We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.

> Numpy does 90% of the work

Numpy and scipy have been the core of a huge amount of my optimisations. The first question I try and ask is

"Could this be solved with matrix multiplications and summing?"

Often the answer is "yes" and allows you to group a huge amount of calculations all together, and use the heavily optimised code available numpy/scipy.

I recently swapped out something that was running at about 100 rows calculated/second to about half a million in about 0.2s.

Re: Why I Still Use Python for High Performance Scientific Computing

#86

Post by andreasvc »

dagw wrote: post #79
harshnisar wrote: post #76

Earlier quoted context omitted.

Can you explain the context in which .keys() is called often and is not appropriate and the alternative?

.keys() returns a list (in python2) so if you write for k in dict.keys(): ... then python first builds a list of all the keys, loops through them and then throws away the list. If the dict is large, this can be quite expensive. The correct way is to either use .iterkeys() which returns an iterator which generates the keys one at a time, or simply iterate directly over the dict, saving you need to first copy all the k…

The easiest is to never use .keys() or .iterkeys(), and always iterate over the bare dict:

    for k in dict:
        ...

    if k in dict:
        ...
If you do need a list of keys, list(dict) has the advantage of working in both Python 2 and 3.

Re: Why I Still Use Python for High Performance Scientific Computing

#87

Post by JustSomeNobody »

wesm wrote: post #21
boulos wrote: post #18

If Numpy, Pandas, etc. were wrappable from JavaScript this could have easily been titled "Why I use Node.js for High Performance Scientific Computing". The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!". I agree some of the tooling and niceties are beyond a doubt best in breed…

Completely bizarre attitude (creator of pandas here).

Thank you for pandas.

Re: Why I Still Use Python for High Performance Scientific Computing

#88

Post by JustSomeNobody »

sukulaku wrote: post #43
wesm wrote: post #21

Earlier quoted context omitted.

Completely bizarre attitude (creator of pandas here).

Why is it bizarre? His point is that the article shouldn't make it sound like "Python is fast", because the speed actually comes from the libraries that have been implemented in C.

The point I think a lot of people are missing is that we've always used optimized libraries no matter what language we've used. C devs would use hand optimized assembly when speed was critical. There's no perfect language that just does everything.

Re: Why I Still Use Python for High Performance Scientific Computing

#89

Post by JupiterMoon »

pjc50 wrote: post #50
olau wrote: post #37

Earlier quoted context omitted.

"...and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code." True. But this assumes that time is not a constraint. I think you need to think of it this way (as a thought experiment): you start two programmers off, one in C and one in Python, both with a vague understanding of how to solve the problem and approximately the same skill level. Then after X ho…

Absolutely. For this kind of one-shot scientific computing, the only way the C programmer can win is if the Python programmer is sitting on their hands for weeks waiting for their program to run.

Unless (1) the relevant library happens to be written in C and there are no python bindings and the problem is simple OR (2) there is an existing solution which is 95% complete in C and one needs to write from scratch in python. I've never come across situation (1) with a new project. Situation (2) is quite common.

Re: Why I Still Use Python for High Performance Scientific Computing

#90

Post by nanoojaboo »

cygnus_a wrote: post #9
msellout wrote: post #2

Summary in the conclusion: "The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]

Actually, even the subsection headings in bold give a very succinct summary: - Python has easy development ( https://xkcd.com/353/ ) - Great libraries (ie, free matlab) - Cython for efficiency via C - The algorithms themselves determine speediness (ie numerical methods)

The creator of Golang has said that most important is to pick the correct data structure and then choosing the optimal algorithm will be almost obvious
Post reply on HN