Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

71–80 of 336 posts

Re: Python’s Weak Performance Matters

#71

Post by banachtarski »

inciampati wrote: post #68
luispedrocoelho wrote: post #36

Earlier quoted context omitted.

OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

If you write more C++ than python, it will have a lower TimeToWriteCode. Despite having spent years writing python I don't find it any more productive than C++. C++11 has all the nice features you might expect from python with the only drawback being the lack of a REPL.

+1 people shouldn't overlook things that are bundled in the C++ stdlib now (chrono, random, thread, algorithm, mutex, containers, etc)

Re: Python’s Weak Performance Matters

#72

Post by joshuamorton »

pjmlp wrote: post #60
joshuamorton wrote: post #25

Earlier quoted context omitted.

While I'm more of a pythonist than a C-ist, hearing "1000x speedup" and "line for line" to me implies that you aren't writing idiomatic python. Idiomatic python is (often) faster than not, and (often) more difficult to translate to lower level languages. As a simple example, list-comprehensions are faster than loops, and can't be line for line translated into C++.

Depends, I bet they can be easily translated to some LINQ like implementation in C++17. List comprehensions are just syntax sugar for map/filter/fold.

A jaunt through SO didn't give anything reasonable.

This is what SO had to offer: https://stackoverflow.com/questions/36339533/how-to-generate..., and a quick check of new C++17 features didn't show any that would obviously improve on that.

There's nothing implicitly stopping you from doing some macro magic to implement it, but its not there naturally.

Re: Python’s Weak Performance Matters

#73

Post by banachtarski »

thinkpad20 wrote: post #4
matt_wulfeck wrote: post #2

> The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.

How so? Haskell has a very high performance ceiling.

It really doesn't... I think it's "better" but I definitely wouldn't classify it as high.

Re: Python’s Weak Performance Matters

#74

Post by loeg »

banku_brougham wrote: post #46

I’d like to thank the author for sharing a very practical view of problem solving in the data science space. Can I suggest julia? Its very easy to understand coming from python, and performant code can be had usually in easy to read implementation of the expressions in whatever paper you are basing your work upon.

Anytime someone brings up Julia, I think of Dan Luu's review of the language: https://danluu.com/julialang/

Re: Python’s Weak Performance Matters

#75

Post by Animats »

ChrisSD wrote: post #11

I don't think it's true to say that Python's core developers are uninterested in performance. Speeding up Python is a hard problem. He mentions PyPy but even that has only managed modest performance gains in some areas (and not without tradeoffs). He suggests JavaScript as a comparison but doesn't elaborate on how they're comparable beyond the superficial (they're both dynamic scripting languages). I get that he's fr…

Python's little tin god really likes his CPython implementation being the One True Python. Python does the things that are easy to do in an interpreter where everything is a dictionary, and avoids things which are hard to do in that environment. In Python, you can store into any variable in any thread from any other thread. You can replace code being executed in another thread. Even Javascript doesn't let you do that. This functionality is very rarely used, and makes it really hard to optimize Python.

(And no, calling C whenever you need to go fast is not a solution. Calling C from Python is risky; you have to maintain all the invariants of the Python system, manually incrementing and decrementing reference counts, and be very careful about not assuming things don't change in the data structures you're looking at. This is not trivial.)

A generation ago, Pascal had the same problem. Wirth had an elegant recursive-descent compiler that didn't optimize. He insisted it be the One True Compiler, and managed to get the ISO standard for Pascal to reflect that. The decline of Pascal followed, although Turbo Pascal for DOS, a much more powerful dialect, had a good run, and Delphi still lives on.

Re: Python’s Weak Performance Matters

#76

Post by e12e »

civility wrote: post #14
hyperion2010 wrote: post #7

I recently discovered that pypy3 can run all my day to day Python code. It has some issues with slightly different behavior from cpython when using threads but other than that I see a 4x speedup on most of my slowest pure python workloads (parsing large rdf files and reserializing them after computing a total order on all their nodes). Huge win for productivity.

> I see a 4x speedup on most of my slowest pure python workloads Heh, only 25..250 X to go. We did a direct line for line translation of some numerically intensive code from Python to C++ and saw a literal 1000X speedup. On other projects, it's been more like 100X slower. That says two things: first Python can be really slow, second, for some programs, Python doesn't really save on lines of code over modern C++. I've…

> Finally, I've always been able to get Cython to parity with C++. However, when I'm done, I wonder what I gained. The C++ isn't that much more complicated than adequately type annotated Cython.

Depends on who needs to work with the code; but sometimes, it might be nice to have an obviously correct (but slow) pure python version, that can share apis and tests with the convoluted "fast enough" version.

Re: Python’s Weak Performance Matters

#77

Post by loeg »

ChrisSD wrote: post #11

I don't think it's true to say that Python's core developers are uninterested in performance. Speeding up Python is a hard problem. He mentions PyPy but even that has only managed modest performance gains in some areas (and not without tradeoffs). He suggests JavaScript as a comparison but doesn't elaborate on how they're comparable beyond the superficial (they're both dynamic scripting languages). I get that he's fr…

PyPy achieves huge performance gains in most long-running code. I don't know what you mean by modest. The main hurdle is slightly longer start-up times while the JIT warms up (comparable to Java).

http://speed.pypy.org/

Re: Python’s Weak Performance Matters

#78

Post by pcx »

munro wrote: post #66

> At the same time, data keeps getting bigger and computers come with more and more cores (which Python cannot easily take advantage of), while single-core performance is only slowly getting better. Thus, Python is a worse and worse solution, performance-wise. PySpark is makes it really easy to take advantage of multiple cores & machines. Most operations I want to do to my data I can find in PySpark's pyspark.sql.fun…

This is true, but using a Distributed system like Spark itself adds a ton of complexity in having to understand and manage it. If one can do something with a set of stateless processes, even if it's more performant, I feel it's a bad idea to use a distributed system instead. Not always, but a good majority of cases that I've seen. I've seen projects where Celery would be enough but instead they chose to use Spark/Storm and never delivered.

Re: Python’s Weak Performance Matters

#79

Post by [deleted] »

indubitable wrote: post #31
woolvalley wrote: post #15

Earlier quoted context omitted.

It has been for the longest time been a closed source MSFT only thing. It wasn't open source and running on linux was a second class citizen. Not sure if it is still a second class citizen. It's also seen as something fairly heavyweight to write things in, such as Java. You probably don't see it used much for the same reasons why Java isn't used.

Why do you think it's been closed source with minimal cross platform effort? Most of everything from Microsoft related to C# is open source. This includes their compiler, the runtime, and the framework and libraries. And it's been this way since 2014. The .NET standard itself has always been open and cross platform implementations like Mono go back to 2004! I'm not sure what you mean by 'heavyweight.' Its performance…

[deleted]

Re: Python’s Weak Performance Matters

#80

Post by myopicgoat »

loeg wrote: post #74
banku_brougham wrote: post #46

I’d like to thank the author for sharing a very practical view of problem solving in the data science space. Can I suggest julia? Its very easy to understand coming from python, and performant code can be had usually in easy to read implementation of the expressions in whatever paper you are basing your work upon.

Anytime someone brings up Julia, I think of Dan Luu's review of the language: https://danluu.com/julialang/

From 4 years ago... a lot has changed since. Consider this recent post for example: https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-...
Post reply on HN