Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

151–160 of 336 posts

Re: Python’s Weak Performance Matters

#151

Post by Erwin »

Rotareti wrote: post #118

I use a lot of Python for web stuff and I haven't been in a situation where Python itself was the performance bottleneck. I always thought that when you run into a situation where Python is the bottleneck, you replace the critical bits with something like C/C++/Rust. Following this approach, you would get the best of both worlds: rapid proof-of-concept / time-to-marked with the option to improve performance critical…

It requires that your code is architected so performance critical sections of Python can move into C etc. Let's say that your code creates a complex object tree from some configuration settings, and executes Python methods and code from all over it, using heavy OO. That is difficult to move to C++ as your performance is spent on Python bookkeeping -- you are calling methods and thus looking things up in dictionary, you are modifying fields and also looking up more thing in dictionaries, increasing decreasing refcounts etc.

If you have a million 32-bit numbers that you current run Python code on, great, you don't have to convert Python objects to C at all.

Re: Python’s Weak Performance Matters

#152

Post by autokad »

scribu wrote: post #149
passive wrote: post #137

This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…

I think the article makes a lot more sense if you consider it in the context of "Python for data science". In the last few years, there's been a lot of hype about replacing other number crunching solutions (R, SPSS, even Matlab) with the Python ecosystem of tools (Pandas, SciPy, etc.).

i dont seem to follow. if you are doing data science, all the bottle necked stuff will be running in numpy or pyspark. Choosing python over R, SPSS, Matlab usually doesnt come down to which one is faster, and R as far as i know is at least not vastly superior in speed.

Re: Python’s Weak Performance Matters

#153

Post by lpmay »

I don't see it mentioned here, but the dask library looks like a promising solution,. It has ways to handle to these kinds of large datasets, and efficiently schedule computations that don't fit a numpy model. Worth a look.

Re: Python’s Weak Performance Matters

#154

Post by chestervonwinch »

scribu wrote: post #149
passive wrote: post #137

This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…

I think the article makes a lot more sense if you consider it in the context of "Python for data science". In the last few years, there's been a lot of hype about replacing other number crunching solutions (R, SPSS, even Matlab) with the Python ecosystem of tools (Pandas, SciPy, etc.).

I don't understand what you mean about "hype about replacing ... with Python". How is it hype when the majority of people already use Python (see link)?

https://www.kaggle.com/surveys/2017

Re: Python’s Weak Performance Matters

#155

Post by willtim »

imtringued wrote: post #150
bjoli wrote: post #108

Earlier quoted context omitted.

All comparisons of Haskell and C I have seen where Haskell has been near the performance of C(++) has always compared a highly optimised Haskell program to a moderately fast c program. I have found that most managed languages generally come within 2-5 times slower than C and C++. Which is good enough for me.

The JVM is fast enough for me but for some reason any JVM project requires hundreds of megabytes of RAM even for the simplest things. cpp webserver + sqlite database + cygwin overhead? 9MB RAM lua worker + websocket client? 4MB RAM JVM server + static html page + websocket relay + kurento api? 500MB RAM With the exception of the cpp webserver none of these tasks are CPU or memory intensive yet the JVM is still off by…

Have you tried specifying a smaller maximum heap size? The biggest memory issue IMHO, is that Java and the JVM do not specialise any "generic" code and so there can be a lot of unnecessary boxing going on.

Re: Python’s Weak Performance Matters

#156

Post by b0rsuk »

Is writing extensions a lost art? I read a few blog posts about speeding up Python and Ruby with Rust extensions. This should enable rewriting only the slow parts. Later, you could replace more of it if needed. Is writing extensions so very problematic in practice?

I know Go has runtime issues making it not very good for mixing with other languages, so it often encourages rewriting the whole application in it.

Re: Python’s Weak Performance Matters

#158

Post by xyhopguy »

AllegedAlec wrote: post #111

Interesting article. I mostly agree. OP, could I ask a question? You mention 1TB files. Why do you guys at embl not use a database for this sort of stuff? I'd figure that with some proper indexing, I figure you could see pretty decent speedups just from that already.

Bioinformatics is perpetually ten years behind. The de facto standard for sequencing data is effectively a stripped down bzipped plain text file. It's madness.

Re: Python’s Weak Performance Matters

#159

Post by scribu »

chestervonwinch wrote: post #154
scribu wrote: post #149

Earlier quoted context omitted.

I think the article makes a lot more sense if you consider it in the context of "Python for data science". In the last few years, there's been a lot of hype about replacing other number crunching solutions (R, SPSS, even Matlab) with the Python ecosystem of tools (Pandas, SciPy, etc.).

I don't understand what you mean about "hype about replacing ... with Python". How is it hype when the majority of people already use Python (see link)? https://www.kaggle.com/surveys/2017

The fact that a majority of people use Python for data crunching today doesn't prove that there wasn't hype in the past.

I'm not trying to say it's wrong, just that it's become a very visible niche for the language.

Re: Python’s Weak Performance Matters

#160

Post by vgy7ujm »

dTal wrote: post #144
carlmr wrote: post #138

Earlier quoted context omitted.

Exactly, the ecosystem is the only reason IMHO why Python is "easy". I find I'm much quicker at developing C#/F# if the library is available on NuGet (which is often, but not always the case). They're both reasonably fast, too. Python only has a very large community and thus ecosystem to offer.

It can't be the only reason, or how would it ever have attracted such an ecosystem in the first place? Especially with its performance characteristics, and lack of corporate backing. No, Python was invented at a time when its closest competitor was Perl - and you need only compare typical Perl with typical Python to appreciate that Python really was a usability revelation. But that was nearly 30 years ago. I do think…

IME Perl can be more readable if you are experienced in Perl.
Post reply on HN