Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

51–60 of 336 posts

Re: Python’s Weak Performance Matters

#51

Post by alphaalpha101 »

ChrisSD wrote: post #11

I don't think it's true to say that Python's core developers are uninterested in performance. Speeding up Python is a hard problem. He mentions PyPy but even that has only managed modest performance gains in some areas (and not without tradeoffs). He suggests JavaScript as a comparison but doesn't elaborate on how they're comparable beyond the superficial (they're both dynamic scripting languages). I get that he's fr…

Python is much more dynamic than Javascript. Python lets you inspect stack frames. Python's metaprogramming features exist, for a start, but are also widely used.

Javascript is a pretty static-feeling language when you come from Python. Most of its 'dynamic' nature can be captured by Typescript's type system i.e. basically returning different types from functions depending on which constant values you pass in. This sort of thing:

    declare function foo(x: 'type1'): Type1;
    declare function foo(x: 'type2'): Type2;
Python is dynamic on a whole 'nother level.

Re: Python’s Weak Performance Matters

#52

Post by pletnes »

luispedrocoelho wrote: post #36
matt_wulfeck wrote: post #2

> The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.

OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

That’s easy with python, too, in a lot of number crunching cases. Numpy with MKL will use all your cores, as will e.g dask and other libraries built on numpy. Farming out embarassingly parallel work to threads or processes is also easy.

Re: Python’s Weak Performance Matters

#53

Post by marmaduke »

His example of a function which is unreadable, is pretty typical. It still might be slower than a tight loop in C, but it’s only unreadable the first time you write something like that.

That said, Numba would be a natural tool here.

Re: Python’s Weak Performance Matters

#54

Post by WillReplyfFood »

The problem here is that most language are tailored to easy learning by humans and not to strong performance. If you had a language tailored to strong performance, it would force you to bundle together seemingly unrelated data into structures used in the main hot codepath of the core algorithm.

It would then force you to specify data delivery routes and processing- and would craft a cache optimal process loop dependent on the used architecture. The result would seem like a enormous while loop, that takes seemingly arbitrary number of for loops to preprocess data, glued together in strange overleafing unions to shove the endresult to the main process algorithm.

This would be the most optimal result for a processor- but to write a language to describe this- and compilers to implement this) - the horror.

Re: Python’s Weak Performance Matters

#55

Post by benou »

ivanb wrote: post #47

I have an impression that there are features in Python that give very little programmer productivity but make the language slow. It should be possible to implement a hypothetical FastPython without such features but with great performance gains. Of course it wouldn't be compatible with most of the libraries. I can imagine though that porting most of the libraries to FastPython still would be a manageable task. I wond…

You mean like Cython [1] or RPython [2] from PyPy [3] ? [1] http://cython.org/ [2] https://rpython.readthedocs.io/en/latest/rpython.html [3] http://pypy.org/

Re: Python’s Weak Performance Matters

#57

Post by pjmlp »

acqq wrote: post #40
ohyes wrote: post #32

Earlier quoted context omitted.

“Lies, damn lies and benchmarks” I think is the saying. Python itself is a slow language but it has a lot of fast packages, so it shows poorly when you actually write your benchmark in python. Haskell is a faster language but because it is high level there are more pitfalls you’ll get into if you don’t know the ins and outs of getting fast code out of the compiler. The guys who write fast benchmark code aren’t ‘avera…

> Python has a shallower learning curve and an easy way to get “good enough” performance (a bit slower than C). The article we all reply to exactly claims that as soon as you don't use e.g. NumPy, it's not "good enough" anymore, and I agree with that. The article also argues that e.g. JavaScript isn't more in the same category with Python, but much faster, even if it's not less dynamic. I think the reason for JavaScr…

Every time this comes up, there are also the good examples of Lisp, Dylan and Smalltalk as languages that as dynamic as Python, while enjoying relatively good JIT compilers.

Re: Python’s Weak Performance Matters

#58

Post by joshuamorton »

luispedrocoelho wrote: post #48
joshuamorton wrote: post #44

Earlier quoted context omitted.

While that may be true, my point is that it is almost certainly possible to make your code go faster than it is already, and also become more readable in the process. And so saying that python is either slow or ugly and unreadable is perhaps an unfair characterization. I may be wrong here. I haven't benchmarked the code in question, but I think that even for the algorithm you're trying to do, with the special casing,…

That may be the case. However, my point is that we started with a rather direct implementation of a formula in a paper. This was very easy to write but took hours on a test set (which we could extrapolate to taking weeks on real data!). Then, I spent a few hours and ended up with that ugly code that now takes a few seconds (and is dominated by the whole analysis taking several minutes, so it would not be worth it eve…

Right, and my point is that you could probably

1. Have gotten similar performance boosts elsewhere, meaning that you wouldn't have needed to refactor this function in the first place (although the implication of a 10000x speedup means that may not be true, although I can absolutely see the potential for 100x speedups in this code, depending on exactly what the input data is)

2. Its likely that there are much more natural ways to implement the function you have in pandas more idiomatically. These would be both clearer and likely equally fast, though possibly faster. (heck, there are even ways to refactor the code you have to make it look a lot like the direct from the paper impl)

In other words, this isn't (necessarily) a case of python having weak performance, its a case of unidiomatic python having weak performance. This is true in any language though. You can write unidiomatic code in any language, and more often than not it will be slower than a similar idiomatic method (repeatedly apply `foldl` in haskell). I'm not enough of an expert in pandas multi-level indexes to say that for certain, but I'd bet there are more efficient ways to do what you're doing from within pandas that look a lot less ugly and run similarly fast.

Granted, there's an argument to be made that the idiomatic way should be more obvious. But "uncommon pandas indexing tools should be more discoverable" is not the same as "python is unworkably slow".

Re: Python’s Weak Performance Matters

#59

Post by pjmlp »

ChrisSD wrote: post #11

I don't think it's true to say that Python's core developers are uninterested in performance. Speeding up Python is a hard problem. He mentions PyPy but even that has only managed modest performance gains in some areas (and not without tradeoffs). He suggests JavaScript as a comparison but doesn't elaborate on how they're comparable beyond the superficial (they're both dynamic scripting languages). I get that he's fr…

Common Lisp, Dylan and Smalltalk are as dynamic as Python, yet they all enjoy of good quality native code compilers (AOT/JIT).

Re: Python’s Weak Performance Matters

#60

Post by pjmlp »

joshuamorton wrote: post #25
civility wrote: post #14

Earlier quoted context omitted.

> I see a 4x speedup on most of my slowest pure python workloads Heh, only 25..250 X to go. We did a direct line for line translation of some numerically intensive code from Python to C++ and saw a literal 1000X speedup. On other projects, it's been more like 100X slower. That says two things: first Python can be really slow, second, for some programs, Python doesn't really save on lines of code over modern C++. I've…

While I'm more of a pythonist than a C-ist, hearing "1000x speedup" and "line for line" to me implies that you aren't writing idiomatic python. Idiomatic python is (often) faster than not, and (often) more difficult to translate to lower level languages. As a simple example, list-comprehensions are faster than loops, and can't be line for line translated into C++.

Depends, I bet they can be easily translated to some LINQ like implementation in C++17.

List comprehensions are just syntax sugar for map/filter/fold.

Post reply on HN