Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

31–40 of 336 posts

Re: Python’s Weak Performance Matters

#31

Post by indubitable »

woolvalley wrote: post #15
indubitable wrote: post #5

Something I often wonder in these sort of discussions is why C# is generally omitted. Its performance is comparable to C++, with none of the trappings. It also does an excellent job of integrating some of the most useful features of functional programming into an imperative language. And multi-processor programming with the language is also incredibly simple. But I think the best part is in programmer time. An anecdo…

It has been for the longest time been a closed source MSFT only thing. It wasn't open source and running on linux was a second class citizen. Not sure if it is still a second class citizen. It's also seen as something fairly heavyweight to write things in, such as Java. You probably don't see it used much for the same reasons why Java isn't used.

Why do you think it's been closed source with minimal cross platform effort? Most of everything from Microsoft related to C# is open source. This includes their compiler, the runtime, and the framework and libraries. And it's been this way since 2014. The .NET standard itself has always been open and cross platform implementations like Mono go back to 2004! I'm not sure what you mean by 'heavyweight.' Its performance is head to head with C++.

Here's the primary repo for .NET goodies: https://github.com/microsoft/dotnet

---

And if anybody downvoting could take a second to actually chime in, it would be illuminating. From my perspective all I see an immense amount of misinformation about the language, and I'm genuinely curious what people may not like about it.

Re: Python’s Weak Performance Matters

#32

Post by ohyes »

zwerdlds wrote: post #8
matt_wulfeck wrote: post #2

> The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.

Not saying you're wrong - you definitely don't get great performance easily in Haskell, but it seems to have better benchmarks than python, by a lot.

“Lies, damn lies and benchmarks” I think is the saying.

Python itself is a slow language but it has a lot of fast packages, so it shows poorly when you actually write your benchmark in python.

Haskell is a faster language but because it is high level there are more pitfalls you’ll get into if you don’t know the ins and outs of getting fast code out of the compiler. The guys who write fast benchmark code aren’t ‘average’ developers.

So in Haskell it’s “the code is slow and I don’t know why” vs python “the code is slow because python is slow, import fast package someone wrote to speed it up.”

All that said, I think Haskell is the better language but you have to put in more effort to get experienced in it before you see returns on the investment. Python has a shallower learning curve and an easy way to get “good enough” performance (a bit slower than C).

The best criticism in the article is of the multi-core deficiency of python’s interpreter. But that’s only briefly touched on. It isn’t a friendly environment to write complicated multi core code.

Re: Python’s Weak Performance Matters

#33

Post by anameaname »

Insanity wrote: post #3

does this not come down to using the right tool for the job? I love writing in Python but would not use it for something where performance matters.

It's hard to tell ahead of time if performance matters. When you hack something out, and are able to get it to work, it inevitably starts to grow features. At some point (around 1000 lines in my experience), Python no longer is fast enough, but the code is difficult to port to something faster.

It would be much better if you didn't have to back track on all the code you wrote. I've been burned by this enough times, that now I am wary of starting anything in Python, because I know it will grow to be bigger, and I will regret having picked Python!

Re: Python’s Weak Performance Matters

#34

Post by indubitable »

pletnes wrote: post #12
indubitable wrote: post #5

Something I often wonder in these sort of discussions is why C# is generally omitted. Its performance is comparable to C++, with none of the trappings. It also does an excellent job of integrating some of the most useful features of functional programming into an imperative language. And multi-processor programming with the language is also incredibly simple. But I think the best part is in programmer time. An anecdo…

The author is a scientist analyzing his data. I never met anyone in that crowd using C#. Are there even any good data science/numerics libs out there? C++ has a lot of number crunching libs, python even more.

This is not my field, but I have used Accord.NET (http://accord-framework.net/) while learning machine learning and have no complaints, but also have little room for comparison. Any C++ library you'd consider as definitive, for some basis to compare? One other thing is that running C++ libraries in C# is, in most cases, quite trivial.

Re: Python’s Weak Performance Matters

#35

Post by willtim »

matt_wulfeck wrote: post #2

> The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.

GHC Haskell is an advanced optimising compiler, which can get very near the speed of C and C++. However, to write fast programs, one must use the right data structures and algorithms. Often this means array-based strings and streaming IO, unfortunately many Haskell textbooks don't tend to cover this.

Re: Python’s Weak Performance Matters

#36

Post by luispedrocoelho »

matt_wulfeck wrote: post #2

> The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.

OP here.

Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode.

Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me.

Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

Re: Python’s Weak Performance Matters

#37

Post by luispedrocoelho »

willtim wrote: post #35
matt_wulfeck wrote: post #2

> The result is that I find myself doing more and more things in Haskell, which lets me write high-level code with decent performance (still slower than what I get if I go all the way down to C++, but with very good libraries). This strikes me as an odd conclusion to come to if speed was the main motivator.

GHC Haskell is an advanced optimising compiler, which can get very near the speed of C and C++. However, to write fast programs, one must use the right data structures and algorithms. Often this means array-based strings and streaming IO, unfortunately many Haskell textbooks don't tend to cover this.

Yep, my Haskell usage is "conduit all the way down".

I even wrote up a few utilities to make use of multiple threads while working at a high level: https://hackage.haskell.org/package/conduit-algorithms-0.0.7...

Re: Python’s Weak Performance Matters

#38

Post by luispedrocoelho »

Insanity wrote: post #3

does this not come down to using the right tool for the job? I love writing in Python but would not use it for something where performance matters.

I think this is part of my argument: as datasets grow faster than single-core CPU speed, performance matters more and more.

Re: Python’s Weak Performance Matters

#39

Post by joshuamorton »

hidenotslide wrote: post #29

I don't find this a very compelling argument. The author doesn't mention any attempts to profile or speed up the code. Specifically with pandas I've found if you aren't careful you can do a lot of unnecessary copying. Not sure if that's what is going on here, but cProfile can help find the bottlenecks.

Seconding this, there are a couple of things that jump out at me as immediately non-optimal, and which together would probably give an order of magnitude speedup.

- Defining compute_diversity inside a double for loop

- `sample1.ix[sample1.index[sample1.index.duplicated()]]` appears overengineered (I think you can just remove the `sample1.index` here (edit: you can't , but I think you could refactor to remove the indexing and reindexing and index resetting, and then you could))

- Depending on the data size, swapping from `[` to `(` everywhere would give a nice speedup just because you no longer need to store everything in memory/swap to disk, whereas in haskell the list comprehensions would be lazy by default. (edit: seeing as the databases downloaded are 12 and 33 GB, and Pandas requires generally 2-3X ram, its likely that there's swapping happening somewhere. I'd bet that using generators would be a big speed boost)

- Overall I think genetic_distance can be significantly simplified, a lot of the index-massaging doesn't look necessary. I could be wrong, but this looks sloppy, and sloppy often implies slower than necessary.

Unfortunately, the provided data files are big enough that I can't easily benchmark on my computer. I can't even fit the dataset in memory!

Re: Python’s Weak Performance Matters

#40

Post by acqq »

ohyes wrote: post #32
zwerdlds wrote: post #8

Earlier quoted context omitted.

Not saying you're wrong - you definitely don't get great performance easily in Haskell, but it seems to have better benchmarks than python, by a lot.

“Lies, damn lies and benchmarks” I think is the saying. Python itself is a slow language but it has a lot of fast packages, so it shows poorly when you actually write your benchmark in python. Haskell is a faster language but because it is high level there are more pitfalls you’ll get into if you don’t know the ins and outs of getting fast code out of the compiler. The guys who write fast benchmark code aren’t ‘avera…

> Python has a shallower learning curve and an easy way to get “good enough” performance (a bit slower than C).

The article we all reply to exactly claims that as soon as you don't use e.g. NumPy, it's not "good enough" anymore, and I agree with that. The article also argues that e.g. JavaScript isn't more in the same category with Python, but much faster, even if it's not less dynamic.

I think the reason for JavaScript's speed vs. Python's slowness is obvious: there were wealthy companies involved, which were, due to competition pressures, motivated to speed up their own JavaScript engines.

To get to the point where Python has similar speeds somebody would have to be motivated enough to invest heavily, and then it could happen. As far as I know, there aren't technical limitations against that.

Post reply on HN