Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

131–140 of 336 posts

Re: Python’s Weak Performance Matters

#131

Post by otorrillas »

OP, I would encourage you take some courses on High-performance computing and, specially, on Architecture Awareness in Programming. These types of courses will help you increasing the performance of your programs by being aware of what's running "under the hood" and ways to "help" the compiler/interpreter better optimisations.

Although it's accurate to say that Haskell or C++ are faster than Python, by having had a quick look on the examples you posted around here, I believe there's still a lot of room to improve (performance-wise) on your Python code that could bring a significant speedup boost.

However, bear in mind that you shouldn't expect Python to be close to a C++ performance unless you start using libraries such as NumPy that are, essentially, written in C/C++.

Re: Python’s Weak Performance Matters

#132

Post by klibertp »

hoschicz wrote: post #115
fermigier wrote: post #45

The go-to solution for speeding up Python code should always be first to use Cython on critical sections of your Python code and tweak your code using type annotations, at least IMHO.

Do type annotations really make any difference to the interpreter? I thought that the interpreter doesn't care about what type a variable is annotated to...

Cython compiles to native library exposing Python extensions API.

Re: Python’s Weak Performance Matters

#133

Post by eesmith »

indubitable wrote: post #119
eesmith wrote: post #107

Earlier quoted context omitted.

You wrote "Things like ADO/ASP/Windows Forms/etc were open sourced back in 2008". Could you explain that further? It appears to contradict this ZDNet article from 2014, http://www.zdnet.com/article/microsoft-to-open-source-more-o... , which says: > Microsoft to open source more of .NET, and bring it to Linux, Mac OS X > Microsoft is porting its server-side .NET stack to Linux and Mac OS X, and is making more of that…

Sure, here's an article from Microsoft describing how to access and view the source code for their library implementation, ASP/ADO/Forms/etc: https://weblogs.asp.net/scottgu/net-framework-library-source... That's from January 2008. This release is specifically what enabled Mono to really go to the next level, more than a decade ago. You can even see the little 'carve out' they made in the license to ensure Mono, in p…

The link you gave is indeed to view the source. It is not, however, an open source license, as characterized by the OSI or DFSG, nor free software as characterized by FSF.

Here is the license - https://referencesource.microsoft.com/license.html . It contains obviously non-open source clauses like:

> "Reference use" means use of the software within your company as a reference, in read only form, for the sole purposes of debugging your products, maintaining your products, or enhancing the interoperability of your products with the software, and specifically excludes the right to distribute the software outside of your company.

Microsoft agrees that (quoting from a quote in my earlier comment) "the source code was available since Rotor but we didn’t use an OSI approved open source license, which made Rotor a non-starter".

Like I said, I know little about this topic. However, I think you do not understand what open source means.

I'm pretty sure that when woolvalley used the phrase "It has been for the longest time been a closed source MSFT only thing", that "closed source" meant "not open source according to OSI or similar guidelines." Not "available in source code form", which is what you seem to think it means.

Re: Python’s Weak Performance Matters

#134

Post by jkabrg »

BerislavLopac wrote: post #117
luispedrocoelho wrote: post #99

Earlier quoted context omitted.

That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in…

The speed could possibly be improved by using map. Also, not related to speed if this is all of the code, but might affect it in a larger programs: you should make sure your file pointers are closed. Something like: with open('interesting.txt') as interesting_file: interesting = {line.strip() for line in interesting_file} with open('data.txt') in data_file: total = sum(int(val) for id, val in map(lambda line: line.sp…

`map` is not going to make it faster. `map` is a loop. Only vectorized code is faster.

Re: Python’s Weak Performance Matters

#136

Post by alecco »

narimiran wrote: post #102

If you would like 10-100x faster performance than Python, but would like to keep the easy-to-read code, give Nim [0] a try. I do all my work in Python, and I've been using Nim in last couple of months - it took me a week or two until I was able to be productive in Nim. Don't expect Python's large ecosystem, nor some Python goodies, but if you're looking for a readable, writable, high-performance post-Python language…

Or just program a module in C or Cython.

Re: Python’s Weak Performance Matters

#137

Post by passive »

This is a weird article at this point in time.

The question it addresses:

"Does Python's performance matter?"

Has always had the answer:

"Sometimes, and you have options for those cases."

The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be.

I don't know of any Python advocates who say it's the right tool for every part of every job. What we will say is it's usually a good "first" tool for every job. Building a system out in Python allows you to get something representative fairly quickly, which helps identify if there are areas where Python alone is not enough.

Re: Python’s Weak Performance Matters

#138

Post by carlmr »

platz wrote: post #126
narimiran wrote: post #102

If you would like 10-100x faster performance than Python, but would like to keep the easy-to-read code, give Nim [0] a try. I do all my work in Python, and I've been using Nim in last couple of months - it took me a week or two until I was able to be productive in Nim. Don't expect Python's large ecosystem, nor some Python goodies, but if you're looking for a readable, writable, high-performance post-Python language…

I mean, if you're willing to give up the ecosystem, there are tons of options out there.

Exactly, the ecosystem is the only reason IMHO why Python is "easy". I find I'm much quicker at developing C#/F# if the library is available on NuGet (which is often, but not always the case). They're both reasonably fast, too. Python only has a very large community and thus ecosystem to offer.

Re: Python’s Weak Performance Matters

#139

Post by loeg »

shele wrote: post #101
loeg wrote: post #74

Earlier quoted context omitted.

Anytime someone brings up Julia, I think of Dan Luu's review of the language: https://danluu.com/julialang/

The way Dan Luu's post is used on HN resembles a thought terminating cliche.

Dan is thorough, and I trust him to make a good faith effort to understand things. If you'd like to refute the arguments and not the messenger, I would love to learn more.

Re: Python’s Weak Performance Matters

#140

Post by proto-n »

luispedrocoelho wrote: post #99
aldanor wrote: post #85

Earlier quoted context omitted.

@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-containe…

That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in…

I've also found that loops with dictionary (or set) lookups are a pain point in python performance. However, this example strikes me as a pretty-obvious pandas use-case:

    interesting = set(line.strip() for line in open('interesting.txt'))
    total=0
    for c in chunks: # im lazy to actually write it
        df = pd.read_csv('data.txt', sep='\t', skiprows=c.start, nrows=c.length, names=['id','val'])
        total += df['val'][df['id'].isin(interesting)].sum()
I'm not exactly sure, but pretty sure that isin() doesn't use python set lookups, but some kind of internal implementation, and is thus really fast. I'd be quite surprised if disk IO wasn't the bottleneck in the above example.
Post reply on HN