Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

241–250 of 336 posts

Re: Python’s Weak Performance Matters

#241

Post by onuralp »

luispedrocoelho wrote: post #99
aldanor wrote: post #85

Earlier quoted context omitted.

@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-containe…

That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in…

Have you checked out scikit-allel? It is fairly comprehensive in terms of calculating basic population stats, and the developer is highly active.

scikit-allel: http://scikit-allel.readthedocs.io/en/latest/index.html

scikit-allel example: http://alimanfoo.github.io/2015/09/21/estimating-fst.html

zarr: https://github.com/zarr-developers/zarr

Re: Python’s Weak Performance Matters

#242

Post by scaryclam »

passive wrote: post #137

This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…

This is a pattern I've had some success with several times now. Create a quick Python implementation for parts of data pipelines and then go back and re-write in Java/Go/C/whatever the best tool is for that bit of the job later when we know where the bottlenecks are.

Re: Python’s Weak Performance Matters

#243

Post by SnowflakeOnIce »

cturner wrote: post #24

Where are the main blowouts in python performance? For example, is it compilation, evaluation overhead, or memory management?

tl;dr: Python's dynamic features add lots of overhead to every operation, and CPython's simple implementation means you pay the overhead even when you don't use the dynamic features.

A few things quickly come to mind, after having maintained a patched version of Python 2.7:

- The dot operator (e.g. `foo.x`) hides a /very complicated/ resolution process that can be /very expensive/. (The documentation about this process also deceptively makes you /think/ you understand how it all works, whereas you probably don't unless you're intimate with the C implementation.)

- Global variables are slower to access than local variables in CPython: the former require hash table operations, whereas the latter are array operations. Global variables can also be of pretty much any type, not just strings, which further complicates how globals are handled.

- `import` statements are idiomatically done at the top-level of a module, and often are used as qualified imports! E.g., `import os` followed by the use of `os.path.join(foo, bar)` later on. This hits the costs of both global variables and the dot operator.

- Other syntactically simple constructs, like indexing, relational operators, `len(foo)`, etc, all support overloading, increasing the complexity of the implementation of these operators.

- CPython has a simple implementation (bytecode interpreter, not really any optimizations), meaning the cost to support overloading and dynamism is /always paid/.

Re: Python’s Weak Performance Matters

#244

Post by smitherfield »

sitkack wrote: post #224
commandlinefan wrote: post #201

Earlier quoted context omitted.

I've been "premature optimization is the root of all evil"ed in more code reviews than I can count - yet when people complain about software, they first complain that it's ugly and then they complain that it's slow. Users don't care what it's written in.

There is fast (optimization) and slow (the most straightforward thing with no consideration to perf) and then not-slow. I think not-slow is a happy medium. Fast code should and probably is, totally unreadable, looking like AlphaGo optimized your program.

>Fast code should and probably is, totally unreadable, looking like AlphaGo optimized your program.

Yes and no; multithreading and optimizing to remove branches can do a number on readability, but I find optimizing for cache locality and compile-time evaluation often makes code more readable. It depends on the language obviously.

Re: Python’s Weak Performance Matters

#245

Post by cbcoutinho »

dahart wrote: post #221

> It is true that programmer time is more valuable than computer time, but waiting for results to finish computing is also a waste of my time (I suppose I could do something else in the meanwhile, but context switches are such a killer of my performance that I often just wait). In film CG production, we had a rule of thumb. If running an interactive program takes longer than about ten seconds, the artist (user) becom…

Somewhat tangential to this, but I remember reading an article on HN about a group creating some web app that was constrained by some size limit (50 KB or so). Groups ended up putting 'dead code' in their projects to guard against other groups taking their space while they were working on their feature. I think this made it so the app never got less than 50...

Did you ever see something similar where devs put in some code as filler to make room for future features they were working on?

Re: Python’s Weak Performance Matters

#246

Post by simonbyrne »

loeg wrote: post #139
shele wrote: post #101

Earlier quoted context omitted.

The way Dan Luu's post is used on HN resembles a thought terminating cliche.

Dan is thorough, and I trust him to make a good faith effort to understand things. If you'd like to refute the arguments and not the messenger, I would love to learn more.

The post is quite old: while the technical arguments certainly had merit at the time, they have largely been addressed (the exception is probably error handling, but his complaint there is more subjective, and I still don't think any language really has a good answer for that one).

As to the community, I'm not exactly sure what happened with Dan (he only has a handful of posts on GitHub and mailing lists, so it seems to have been largely in private emails), but my experience could not have been more different: even from the early days they have been very friendly and helpful.

(disclaimer: I now work for Julia Computing)

Re: Python’s Weak Performance Matters

#247

Post by [deleted] »

passive wrote: post #137

This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…

[deleted]

Re: Python’s Weak Performance Matters

#248

Post by commandlinefan »

sitkack wrote: post #224
commandlinefan wrote: post #201

Earlier quoted context omitted.

I've been "premature optimization is the root of all evil"ed in more code reviews than I can count - yet when people complain about software, they first complain that it's ugly and then they complain that it's slow. Users don't care what it's written in.

There is fast (optimization) and slow (the most straightforward thing with no consideration to perf) and then not-slow. I think not-slow is a happy medium. Fast code should and probably is, totally unreadable, looking like AlphaGo optimized your program.

In theory I think you're correct - you _can_ go crazy overoptimizing code to the point where it's completely unreadable. However, in my experience, it takes a lot of effort to get to this point, especially in modern languages. It seems to me that most developers err in the other direction: whatever works as long as it took the least amount of time to program (although this might well be partly driven by the management "agile" obsession with accounting for how every 15-minute increment of time was spent).

Re: Python’s Weak Performance Matters

#249

Post by cjalmeida »

jerf wrote: post #162
b0rsuk wrote: post #156

Is writing extensions a lost art? I read a few blog posts about speeding up Python and Ruby with Rust extensions. This should enable rewriting only the slow parts. Later, you could replace more of it if needed. Is writing extensions so very problematic in practice? I know Go has runtime issues making it not very good for mixing with other languages, so it often encourages rewriting the whole application in it.

Extensions aren't a total solution, though, which people often sell them as. You have an impedance mismatch between Python and C code, because Python has all of its objects packed in a way that is very strange to C, so you end up essentially deserializing all objects into C, then back out into Python, in a very expensive and allocation-heavy (on both sides) conversion. If you can set up your computation in Python and…

I'm not sure I follow what's hard about writing extensions. You can do it in modern C++ if you will and even old SWIG can generate decent bindings for you if you keep your interface sane.

Re: Python’s Weak Performance Matters

#250

Post by fafhrd91 »

jerf wrote: post #233
fafhrd91 wrote: post #176

Earlier quoted context omitted.

Python extension doesn’t mean C. Rust works perfectly for extensions, it covers a lot of low level c-api integration and it is fast. You can write whole application in rust and use python as a glue language https://github.com/PyO3/pyo3 Pyo3 library gives you ability to work both diractions. Call python code from rust and call rust code from python.

That sounds like one of the "maze of choices" I mentioned, no? And if you're "writing the whole application in Rust and using Python as a glue language", you don't have the problem that this entire discussion is about, which is when you have Python code that is slow. Python as an extension language is a completely different world. Performance problems there are a much less big deal, because you've already got the opt…

My point is, you are not limited with using extensions only for optimizing hot loops, in rust you can write application logic as well. I doubt you should do this in C for example
Post reply on HN