Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

121–130 of 336 posts

Re: Python’s Weak Performance Matters

#121

Post by guitarbill »

ShroudedNight wrote: post #100
acqq wrote: post #40

Earlier quoted context omitted.

> Python has a shallower learning curve and an easy way to get “good enough” performance (a bit slower than C). The article we all reply to exactly claims that as soon as you don't use e.g. NumPy, it's not "good enough" anymore, and I agree with that. The article also argues that e.g. JavaScript isn't more in the same category with Python, but much faster, even if it's not less dynamic. I think the reason for JavaScr…

Having actively worked on a JIT compiler for CPython; in retrospect, JavaScript had a significant advantage over python: the expected requirement of one's JavaScript code to run more-or-less compatibly on a variety of interpreters. So much Python has historically been tied to CPython's specific ideosyncracies that there is significantly more onus on the upstart VM developers to maintain compatibility with paralinguis…

It seems like the downfall of anything non-CPython is either the C API or the GIL, although I'm shocked at how good package support is looking for PyPy now (http://packages.pypy.org). Makes me want to try it again.

Javascript also has the "benefit" of an appalling base library, while the base library that CPython provides is quite large, and growing.

Re: Python’s Weak Performance Matters

#122

Post by aldanor »

luispedrocoelho wrote: post #99
aldanor wrote: post #85

Earlier quoted context omitted.

@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-containe…

That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in…

Could you give a hint of how the data ("sample1", "sample2") looks like, or how to randomly generate it in order to benchmark it sensibly? I guess these are similarly-indexed float64 series where the index may contain duplicates? Maybe you could share a chunk of data (as input to genetic_distance() function) as an example if it's not too proprietary and if it's sufficient to run a micro benchmark.

There's also code in genetic_distance() function that IIUC is meant to handle the case when sample1 and sample2 are not similarly-indexed, however (a) you essentially never use it, since you only pass sample1 and sample2 that are columns of the same dataframe (what's the point then?), and (b) your code would actually throw an exception if you tried doing that.

P.S. I like the part where you've removed the comment "note that this is a slow computation" :)

Re: Python’s Weak Performance Matters

#123

Post by quietbritishjim »

munro wrote: post #66

> At the same time, data keeps getting bigger and computers come with more and more cores (which Python cannot easily take advantage of), while single-core performance is only slowly getting better. Thus, Python is a worse and worse solution, performance-wise. PySpark is makes it really easy to take advantage of multiple cores & machines. Most operations I want to do to my data I can find in PySpark's pyspark.sql.fun…

The original article said that one reason it doesn't matter that pure Python's performance is poor is that you can use numpy (and pandas) to vectorise things, which then has native code performance. It goes on to say that his current problem is that the things he's doing today can't be vectorised with numpy – so that poor performance does matter after all. If he can't even express his code in terms of numpy operations (and other C-based libraries like scipy), I doubt they're going to be expressible in terms of Spark's primitives, which are a considerably smaller subset.

Re: Python’s Weak Performance Matters

#124

Post by wirrbel »

ptype wrote: post #106

I love Python. The language is a joy, the eco system is fantastic. But yes, let’s be honest, if you can not vectorise your code it is slow, and I think that will be its downfall eventually. I’m excited about Julia, I hope it gains popularity and the eco system grows. Until then, and in particular until the data frames story can compete with pandas, it Python with Cython for me, but I’d rather skip the Cython if it wa…

This is maybe more hearsay, I only briefly tried to use Julia.

It is true, Julia has great features for performant code. It is, however, focussing too much on being a matlab competitor in my opinion. It will not be a language that you use to write a "normal" (i.e. non-numeric or CRUD) dynamic website in. My general observation however is, that you need to attract this crow, if you want to have an ecosystem with a variety of tooling. And it is the neat thing about Python (and Haskell).

Re: Python’s Weak Performance Matters

#125

Post by deckiedan »

luispedrocoelho wrote: post #99
aldanor wrote: post #85

Earlier quoted context omitted.

@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-containe…

That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in…

Have you tried using Cython to compile code like the above? Python's sets / maps / reading data etc should be fairly optimised, so Cython might let you bypass boxing counter variables instead using native C ints or whatever.

Also, if the data you're reading is numeric only - or at least non-unicode / character data - you might be able to get a speed boost reading the data as binary not as python text strings.

Re: Python’s Weak Performance Matters

#126

Post by platz »

narimiran wrote: post #102

If you would like 10-100x faster performance than Python, but would like to keep the easy-to-read code, give Nim [0] a try. I do all my work in Python, and I've been using Nim in last couple of months - it took me a week or two until I was able to be productive in Nim. Don't expect Python's large ecosystem, nor some Python goodies, but if you're looking for a readable, writable, high-performance post-Python language…

I mean, if you're willing to give up the ecosystem, there are tons of options out there.

Re: Python’s Weak Performance Matters

#127

Post by Certhas »

ivanb wrote: post #47

I have an impression that there are features in Python that give very little programmer productivity but make the language slow. It should be possible to implement a hypothetical FastPython without such features but with great performance gains. Of course it wouldn't be compatible with most of the libraries. I can imagine though that porting most of the libraries to FastPython still would be a manageable task. I wond…

Julia goes in that direction.

Re: Python’s Weak Performance Matters

#128

Post by coldtea »

biztos wrote: post #98
luispedrocoelho wrote: post #36

Earlier quoted context omitted.

OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

Interesting assertion re: TimeToWriteCode, but I think there's TimeToWriteCode vs. TimeToWriteGoodCode. I'm working on my first serious Python project right now, and I find it's super easy to throw together some code that more or less works; but for solid, readable, documented, properly unit-tested code I hope is production-ready, it's not any faster than Perl or Golang. (Sure, if you're a Python expert it's faster f…

>Interesting assertion re: TimeToWriteCode, but I think there's TimeToWriteCode vs. TimeToWriteGoodCode.

In lots of areas, "good code" doesn't matter much, if at all.

Scientific computing is full of those cases -- you write code to run a few times, and don't care for maintaining it and running it ever again (as long as the results are correct).

Re: Python’s Weak Performance Matters

#129

Post by gaius »

gameswithgo wrote: post #18
thinkpad20 wrote: post #4

Earlier quoted context omitted.

How so? Haskell has a very high performance ceiling.

it does have a high perf ceiling, but the code isn't easy to write when you approach it. Things like C#, F#, Java, Kotlin, Nim, Lua would be more natural things to turn to when you want something "Easy" like python but faster, I think.

I use OCaml when I need something like a fast, type-safe Python

Re: Python’s Weak Performance Matters

#130

Post by klibertp »

marmaduke wrote: post #53

His example of a function which is unreadable, is pretty typical. It still might be slower than a tight loop in C, but it’s only unreadable the first time you write something like that. That said, Numba would be a natural tool here.

I'll buy you a beer if you can derive the original formula from this code.
Post reply on HN