Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

301–310 of 336 posts

Re: Python’s Weak Performance Matters

#301

Post by anaptdemise »

peterwaller wrote: post #109
aldanor wrote: post #85

Earlier quoted context omitted.

@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-containe…

> you write code (like in the example here [1]) that is bound to be slow, just from a glance at it > [1] https://git.embl.de/costea/metaSNV/blob/master/metaSNV_post.... . Given his code you referenced, could you elaborate on what makes it look slow at a glance, and how you might speed it up? :)

ln 221:

    if snp_taxID not in samples_of_interest.keys():#Check if Genome is of interest
Tracing through, looks like samples_of_interest is a dict. `snp_taxID not in samples_of_interest` would make membership check constant time.

Re: Python’s Weak Performance Matters

#302

Post by artwr »

halbritt wrote: post #268
rhodysurf wrote: post #168

Earlier quoted context omitted.

Except Pandas and SciPy use libraries written in CXX or Fortran and not pure python so speed is not really an issue with them usually

pandas.read_csv is kind of abysmally slow, unfortunately. There are a couple of alternatives, but nothing has really taken hold. Dask exists, but not everyone can run a distributed system to read a multi-gigabyte csv.

Have you tried paratext from wiseio? I have had good experiences with it.

Re: Python’s Weak Performance Matters

#303

Post by kankroc »

fermigier wrote: post #45

The go-to solution for speeding up Python code should always be first to use Cython on critical sections of your Python code and tweak your code using type annotations, at least IMHO.

I don't understand why your comment is so low. Cython is the easy-to-implement solution here.

Re: Python’s Weak Performance Matters

#304

Post by dragonwriter »

hoschicz wrote: post #115
fermigier wrote: post #45

The go-to solution for speeding up Python code should always be first to use Cython on critical sections of your Python code and tweak your code using type annotations, at least IMHO.

Do type annotations really make any difference to the interpreter? I thought that the interpreter doesn't care about what type a variable is annotated to...

> Do type annotations really make any difference to the interpreter?

Note the advice was first use Cython (a compiler for a superset of Python), and then tweak as needed with type annotations. Cython’s compiler definitely uses type annotations.

Re: Python’s Weak Performance Matters

#305

Post by marmaduke »

klibertp wrote: post #130
marmaduke wrote: post #53

His example of a function which is unreadable, is pretty typical. It still might be slower than a tight loop in C, but it’s only unreadable the first time you write something like that. That said, Numba would be a natural tool here.

I'll buy you a beer if you can derive the original formula from this code.

I admit I recognized only some general NumPy things like masks, unique, reductions, outer etc: I don’t use Pandas and not sure what the non-Numpy stuff does.

I still don’t think it’s more obscure than equivalent for loops or FP folds or similar.

Re: Python’s Weak Performance Matters

#306

Post by weberc2 »

dragonwriter wrote: post #295
weberc2 wrote: post #181

Earlier quoted context omitted.

Which is error prone and often actually slower depending on calling patterns between C and Python. There are many languages today that offer better ergonomics than Python/C, and a few (like Go) which offer better ergonomics than Python by itself, all while besting it in performance by one or more orders of magnitude. I like Python; I just wish I could say the same for its developers...

> There are many languages today that offer better ergonomics than Python/C Including Cython. I mean, if you've got a C library, sure, interface it with Python; but unless you want something to be called from something else in addition to Python, dropping to C for performance needn't be the default choice in Python; that's the whole reason Cython exists. > and a few (like Go) which offer better ergonomics than Python…

I would consider Cython if the particular bottlenecks were amenable to calling into a lower level language and if the alternative were porting a large application from Python to something else wholesale, but I would probably never start a new application in Python/Cython if there was any chance that performance would ever matter. The alternatives are simply too good.

> I find Go’s ergonomics to be worse than Python but better, mostly, than Java.

This is surprising. I'm a Python developer, and I still prototype new features in Go.

Re: Python’s Weak Performance Matters

#307

Post by cozzyd »

inciampati wrote: post #68
luispedrocoelho wrote: post #36

Earlier quoted context omitted.

OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

If you write more C++ than python, it will have a lower TimeToWriteCode. Despite having spent years writing python I don't find it any more productive than C++. C++11 has all the nice features you might expect from python with the only drawback being the lack of a REPL.

My time to debug code is usually smaller for C++, although I'm not familiar with the python tooling as much.

Re: Python’s Weak Performance Matters

#308

Post by snissn »

jerf wrote: post #236
snissn wrote: post #195

Earlier quoted context omitted.

Disagreeing with you and agreeing with the parent, it sounds like a lost art... Numpy doesn't pack and unpack python data structures, it just uses C structures. Python extensions I've written just use C/C++ data types and only occasionally passes python native types back to python. Python is amazing for developer productivity, but the methods are a bit opaque.

"Python extensions I've written just use C/C++ data types and only occasionally passes python native types back to python." See my other post; was your system basically using Python as an extension language on a system fundamentally implemented in C/C++? That's not the problem this discussion is about, which is when you have a large pile of Python code that is the main component of your system, and has proved to be s…

> was your system basically using Python as an extension language on a system fundamentally implemented in C/C++?

python for the networking and business logic, C/C++ for the data

Re: Python’s Weak Performance Matters

#310

Post by deathanatos »

fafhrd91 wrote: post #286
deathanatos wrote: post #262

Earlier quoted context omitted.

PyO3 is a fork of rust-cpython, which has a nasty abort issue[1] which is unfortunately a show-stopper for me. It isn't clear to me if PyO3 is also affected by this issue. [1]: https://github.com/dgrunwald/rust-cpython/issues/59

Pyo3 is not affected by this issue. pyo3 compiles in c-api interface, it doesn't use separate libs for that (python27-sys)

> Pyo3 is not affected by this issue. pyo3 compiles in c-api interface, it doesn't use separate libs for that (python27-sys)

At some point, it must use a separate "lib" for that. Some of the core functions in the Python API exist in the Python binary; it is debatable whether one considers that a "separate lib", or not, but is isn't possible to compile it into your binary. (Or there would be two of whatever you decide to compile in, and that would be problematic.)

I looked into it, since you said it was not be affected. PyO3's own README notes that it is affected by the issue; it lists the same proposed solution as the bug against rust-cpython does. While the solution "works", in the sense that you can build a working module from it, the problem with the solution is that the ergonomics of it are terrible; my understanding is that it completely prevents one from being able to `cargo build`.

That said, I was not aware of either `cargo rustc` or `setuptools-rust`; at the time I was looking into it, setuptools lacked the necessary support to implement `setuptools-rust`, so that's nice to see that that has finally occurred. `cargo rustc` alleviates much of the concern around the ergonomics of building the extension, though that'll still be fun to explain to coworkers. The combination of all that would seem to imply that building Rust extensions might finally be somewhat feasible.

Post reply on HN