Where are the main blowouts in python performance? For example, is it compilation, evaluation overhead, or memory management?
A good writeup is here: http://faster-cpython.readthedocs.io/notes_2017.html
Python’s Weak Performance Matters
41–50 of 336 posts
Re: Python’s Weak Performance Matters
#42There’s a project to plug different JIT compilers into CPython, so there’s hope. https://github.com/Microsoft/Pyjion/blob/master/README.md Also, I’ve more than once seen cpython beat C++/Fortran since it’s easier to do the right algo/datastructure things, plus numpy is more optimized than most «amateur» C loop-over-arrays. That being said, faster python is always welcome.
Honestly, NumPy is gonna be hard to beat even for someone knowledgeable in certain use cases, especially ones where the overhead in Python is trumped by time spent in library calls. It's the same reason that it's hard to beat MATLAB or Mathematica in cases they are optimized for despite being relatively slow languages. They are calling some of the most heavily optimized libraries in existence (e.g., BLAS) and using h…
"spending an hour figuring out what arcane incantation I need to pass to np.einsum to get the operation I want"
Yes, I have also had this experience and I hate how in the end, the code is very hard to read, while the for loop was probably trivial.
Re: Python’s Weak Performance Matters
#43I don't find this a very compelling argument. The author doesn't mention any attempts to profile or speed up the code. Specifically with pandas I've found if you aren't careful you can do a lot of unnecessary copying. Not sure if that's what is going on here, but cProfile can help find the bottlenecks.
Seconding this, there are a couple of things that jump out at me as immediately non-optimal, and which together would probably give an order of magnitude speedup. - Defining compute_diversity inside a double for loop - `sample1.ix[sample1.index[sample1.index.duplicated()]]` appears overengineered (I think you can just remove the `sample1.index` here (edit: you can't , but I think you could refactor to remove the inde…
You are commenting on the variant of the code that is fast enough that it doesn't matter.
Re: Python’s Weak Performance Matters
#44Earlier quoted context omitted.
Seconding this, there are a couple of things that jump out at me as immediately non-optimal, and which together would probably give an order of magnitude speedup. - Defining compute_diversity inside a double for loop - `sample1.ix[sample1.index[sample1.index.duplicated()]]` appears overengineered (I think you can just remove the `sample1.index` here (edit: you can't , but I think you could refactor to remove the inde…
You are commenting on the variant of the code that is fast enough that it doesn't matter.
While that may be true, my point is that it is almost certainly possible to make your code go faster than it is already, and also become more readable in the process.
And so saying that python is either slow or ugly and unreadable is perhaps an unfair characterization. I may be wrong here. I haven't benchmarked the code in question, but I think that even for the algorithm you're trying to do, with the special casing, that function could be significantly simplified.
Edit: I'd be curious to see example data that is passed into this function.
Re: Python’s Weak Performance Matters
#45The go-to solution for speeding up Python code should always be first to use Cython on critical sections of your Python code and tweak your code using type annotations, at least IMHO.
Re: Python’s Weak Performance Matters
#46I’d like to thank the author for sharing a very practical view of problem solving in the data science space.
Can I suggest julia? Its very easy to understand coming from python, and performant code can be had usually in easy to read implementation of the expressions in whatever paper you are basing your work upon.
Re: Python’s Weak Performance Matters
#47I have an impression that there are features in Python that give very little programmer productivity but make the language slow. It should be possible to implement a hypothetical FastPython without such features but with great performance gains. Of course it wouldn't be compatible with most of the libraries. I can imagine though that porting most of the libraries to FastPython still would be a manageable task. I wonder if such projects had been attempted.
Re: Python’s Weak Performance Matters
#48Earlier quoted context omitted.
You are commenting on the variant of the code that is fast enough that it doesn't matter.
While that may be true, my point is that it is almost certainly possible to make your code go faster than it is already, and also become more readable in the process. And so saying that python is either slow or ugly and unreadable is perhaps an unfair characterization. I may be wrong here. I haven't benchmarked the code in question, but I think that even for the algorithm you're trying to do, with the special casing,…
That may be the case. However, my point is that we started with a rather direct implementation of a formula in a paper. This was very easy to write but took hours on a test set (which we could extrapolate to taking weeks on real data!).
Then, I spent a few hours and ended up with that ugly code that now takes a few seconds (and is dominated by the whole analysis taking several minutes, so it would not be worth it even if you could potentially make this function take zero time).
Maybe with a few more hours, I could get both readability and speed, but that is not worth it (at this moment, at least).
*
The comment about the benchmark data being large is exactly my point: as datasets are growing faster than CPU speed, low-level performance matters more than it did a few years ago (at least if you are working, as I am, with these large data).
Re: Python’s Weak Performance Matters
#49I don't find this a very compelling argument. The author doesn't mention any attempts to profile or speed up the code. Specifically with pandas I've found if you aren't careful you can do a lot of unnecessary copying. Not sure if that's what is going on here, but cProfile can help find the bottlenecks.
Seconding this, there are a couple of things that jump out at me as immediately non-optimal, and which together would probably give an order of magnitude speedup. - Defining compute_diversity inside a double for loop - `sample1.ix[sample1.index[sample1.index.duplicated()]]` appears overengineered (I think you can just remove the `sample1.index` here (edit: you can't , but I think you could refactor to remove the inde…
Good point about pandas dataframes taking up extra space, and the solution of using chunking/generators. 33 GB is what Wes McKinney would call "medium data": https://twitter.com/wesmckinn/status/413159516096585729
The problem is libraries that works fine when everything fits in RAM start breaking down if you aren't careful. Not really python speed issue, but you lose some of the tools you relied on previously.
Re: Python’s Weak Performance Matters
#50There’s a project to plug different JIT compilers into CPython, so there’s hope. https://github.com/Microsoft/Pyjion/blob/master/README.md Also, I’ve more than once seen cpython beat C++/Fortran since it’s easier to do the right algo/datastructure things, plus numpy is more optimized than most «amateur» C loop-over-arrays. That being said, faster python is always welcome.
Honestly, NumPy is gonna be hard to beat even for someone knowledgeable in certain use cases, especially ones where the overhead in Python is trumped by time spent in library calls. It's the same reason that it's hard to beat MATLAB or Mathematica in cases they are optimized for despite being relatively slow languages. They are calling some of the most heavily optimized libraries in existence (e.g., BLAS) and using h…
Sure, I’ve seen C++ experts beat by numpy, i.e BLAS/LAPACK. A lot of people don’t use BLAS from C++ either, which would improve the performance on that side, too.