Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

Post reply on HN
Print view View original

61–70 of 158 posts

Re: Why I Still Use Python for High Performance Scientific Computing

#61

Post by fengwick3 »

dagw wrote: post #29
cygnus_a wrote: post #9

Earlier quoted context omitted.

Actually, even the subsection headings in bold give a very succinct summary: - Python has easy development ( https://xkcd.com/353/ ) - Great libraries (ie, free matlab) - Cython for efficiency via C - The algorithms themselves determine speediness (ie numerical methods)

The algorithms themselves determine speediness This is so important I wish people would focus more on it. I recently rewrote some Javascript code in (pure) python and got a good 2 orders of magnitude speed up on large inputs just by picking the right data structures and replacing an O(n^3) nested loop with an O(n log n) approach.

Yes, this is actually one of the rare moments where I find my past background in competitive programming helpful. There's a habit I've picked up of constantly running through techniques to speed up whenever I'm coding. Not necessarily the most efficient way to code, but it's a habit difficult to change.

Re: Why I Still Use Python for High Performance Scientific Computing

#62

Post by dagw »

kfk wrote: post #54

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking f…

one wonders if the developers will be able to find the support they need in case any issue arise from a free library.

Continuum Analytics is founded by the creator of Numpy and employs many leading python developers, and they offer support contracts for basically the entire Python/Numpy data analysis stack.

Re: Why I Still Use Python for High Performance Scientific Computing

#63

Post by anon4 »

cygnus_a wrote: post #9
msellout wrote: post #2

Summary in the conclusion: "The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]

Actually, even the subsection headings in bold give a very succinct summary: - Python has easy development ( https://xkcd.com/353/ ) - Great libraries (ie, free matlab) - Cython for efficiency via C - The algorithms themselves determine speediness (ie numerical methods)

Let me just add Numba to the list. It compiles numpy-using code to native via llvm and can remove temporary arrays in the process.

I tried a very simple toy program the other day and while I had to write some things slightly un-pythonically (it can't deal with syntax like a[:] = b+c yet), it performed practically as good as hand-written C code.

Re: Why I Still Use Python for High Performance Scientific Computing

#64

Post by dagw »

pen2l wrote: post #22

This may be a liiiitle bit off-topic, but I really need to get it off my chest: Python for high-performance scientific computer works beautifully ... it's a dream. Scipy/numpy, matplotlib, pandas, ipython. They're all unbelievably awesome. It all just works. Except , when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible. So…

FUD. I've been using Python on windows for years and between Anaconda and Christoph Gohlke's python packages and I've yet to run into something that didn't just work.

Re: Why I Still Use Python for High Performance Scientific Computing

#65

Post by anon4 »

msellout wrote: post #2

Summary in the conclusion: "The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]

I agree with his conclusion that Python has great tools to help performance, but the GIL makes it impossible to effectively make use of separate cores, so I have to wonder just how badly the Java version was written. Properly written high-performance Java (store all data in contiguous arrays, do away with all the OO crap, reuse objects like crazy) should perform as fast, or faster than optimized Python, and you can scale it to as many threads as you like.

Re: Why I Still Use Python for High Performance Scientific Computing

#66

Post by lqdc13 »

anon4 wrote: post #65
msellout wrote: post #2

Summary in the conclusion: "The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]

I agree with his conclusion that Python has great tools to help performance, but the GIL makes it impossible to effectively make use of separate cores, so I have to wonder just how badly the Java version was written. Properly written high-performance Java (store all data in contiguous arrays, do away with all the OO crap, reuse objects like crazy) should perform as fast, or faster than optimized Python, and you can s…

Python has MPI4PY and Multiprocessing. Not sure why you need threads in the first place. Also, many matrix operations autoscale to all cores when compiled against Atlas.

Finally, you can release the GIL with Cython: http://docs.cython.org/src/userguide/parallelism.html

Re: Why I Still Use Python for High Performance Scientific Computing

#67

Post by sukulaku »

wesm wrote: post #46
sukulaku wrote: post #43

Earlier quoted context omitted.

Why is it bizarre? His point is that the article shouldn't make it sound like "Python is fast", because the speed actually comes from the libraries that have been implemented in C.

By bizarre I mean impractical and unhelpful. What's the point of programming at all if we cannot leverage abstractions to make ourselves more productive? I believe what the article says is that "Python has tools that enable a savvy user to achieve better results with less effort". Python is extremely popular in HPC settings (including supercomputers) for this reason. I see nothing disingenuous.

Well, the word "bizarre" has a commonly understood meaning, but you somehow decided to use it to mean something completely different. That's a bit bizarre :P

But the article is titled: "Why I Still Use Python for High Performance Scientific Computing", and it gives the impression that Python - the language - is fast enough for HPSC.

In reality, the reason why he "still" uses Python is that the libraries are fast enough for HPSC. But that's not what people see when reading posts like this.

The message they see is that "Python is fast", not that some of its 3rd-party libraries are fast.

But I should probably stop repeating myself here.

Re: Why I Still Use Python for High Performance Scientific Computing

#68

Post by pbowyer »

lqdc13 wrote: post #41
pbowyer wrote: post #38

> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast. What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io and https://tideways.io , it's felt a step backwards. I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree an…

cProfile takes a while to learn how to use well. What didn't you like about line_profiler? Here's a good guide on how to write fast(ish) code in Python: https://wiki.python.org/moin/PythonSpeed/PerformanceTips Generally, the best strategy for me has been to use NumPy wherever possible and to avoid creating many complex objects. Best to use built in dicts or tuples for things that store data. Thus the only time I run…

> cProfile takes a while to learn how to use well.

I bet :) I have only scratched the surface, that's for sure.

> What didn't you like about line_profiler?

In this case I was trying to profile the overall simulation codebase to find the slow spots (rather than guess) - a simulation that takes 60 minutes to run. line_profiler wasn't great at giving digestible results from that - and I haven't worked out how to write all output (across multiple modules) to file, without specifying the file in each decorator.

I've started to break the codebase down into 'tests' to measure each algorithm separately, and will give line_profiler another go then.

Re memory_profiler, while the mprof command showed all peaks, the line by line output only showed the result after executing each line - so when 4GB of RAM disappeared in a skimage call, only to be released at the end - it wasn't reflected in the output. Which is tricy when trying to reduce overall memory usage.

Re: Why I Still Use Python for High Performance Scientific Computing

#69

Post by kfk »

cpbotha wrote: post #60
kfk wrote: post #54

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking f…

I have first-hand experience with a BI-ish system, squarely targeted at the enterprise and doing quite well there, that we wrote using Django and a whole list of open source components. We did run into some resistance initially, because our stack is almost the opposite in every way of what our enterprise colleagues are used to. However, our development velocity, especially around analytical features and just in gener…

It would be interesting to hear about your experience, did you do a write up somewhere or could I email you with few questions?

My project would be to put different data sources together + to allow users to upload their own structured data via Excel (think financial estimates). The current system has about 450 users, the next might have much more depending if it gets extended to other divisions.

Re: Why I Still Use Python for High Performance Scientific Computing

#70

Post by pbowyer »

anon4 wrote: post #63
cygnus_a wrote: post #9

Earlier quoted context omitted.

Actually, even the subsection headings in bold give a very succinct summary: - Python has easy development ( https://xkcd.com/353/ ) - Great libraries (ie, free matlab) - Cython for efficiency via C - The algorithms themselves determine speediness (ie numerical methods)

Let me just add Numba to the list. It compiles numpy-using code to native via llvm and can remove temporary arrays in the process. I tried a very simple toy program the other day and while I had to write some things slightly un-pythonically (it can't deal with syntax like a[:] = b+c yet), it performed practically as good as hand-written C code.

If your code falls within the subset it supports :) I've not yet got it to run our code in nopython mode - I think the latest problem is a function expecting a function as an argument, but the error messages aren't helpful or enlightening, even with DEBUG turned on.
Post reply on HN