Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

181–190 of 336 posts

Re: Python’s Weak Performance Matters

#181
post #136

If you would like 10-100x faster performance than Python, but would like to keep the easy-to-read code, give Nim [0] a try. I do all my work in Python, and I've been using Nim in last couple of months - it took me a week or two until I was able to be productive in Nim. Don't expect Python's large ecosystem, nor some Python goodies, but if you're looking for a readable, writable, high-performance post-Python language…

Or just program a module in C or Cython.

Which is error prone and often actually slower depending on calling patterns between C and Python. There are many languages today that offer better ergonomics than Python/C, and a few (like Go) which offer better ergonomics than Python by itself, all while besting it in performance by one or more orders of magnitude.

I like Python; I just wish I could say the same for its developers...

Re: Python’s Weak Performance Matters

#182
post #89

Earlier quoted context omitted.

From 4 years ago... a lot has changed since. Consider this recent post for example: https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-...

The linked article doesn't attempt to refute the significant claims from Dan's article (which has an update from a year later, so 2015, at the bottom): 1. That the language is (was) undertested, and as a result, full of easy to run into bugs 2. The language makes it easy to ignore errors 3. APIs are inconsistent 4. The head branch isn't kept build-clean (i.e., often it fails to build) 5. Code is often undocumented, w…

(1) Test coverage has dramatically improved. (2) Improved, and to some degree a matter of taste (Julia is not statically checked, so some errors can only be discovered at runtime) (3) There has been significant refactoring since 2014 to improve consistency, especially in the recent push toward 1.0. (4) Developers are much more disciplined now, and just about everything goes through CI. (5) Docs are much, much better for both users and developers. For the latter, see https://docs.julialang.org/en/latest/devdocs/ast/ and others in that section (none of which existed in 2014, IIRC).

> And bonus community problem:

I can't comment on private discussions to which I was not a party, but people can search the archived google groups discussions (julia-users) for the blog author's name to read the public threads.

That said, the core of the concern, as I understand it, was that these perceived issues would put a damper on growth. Young languages, especially, need growth to outpace attrition to maintain viability (ecosystem, broad testing and platform support, etc.). Three years on, with sustained user-base growth, increased funding, and Julia having been used for a number of articles in high-impact journals (e.g. Nature) -- I think this concern is somewhat less pressing. See https://pkg.julialang.org/pulse.html for a snapshot of the sustained ecosystem growth, and https://discourse.julialang.org to get a sense of the breadth, depth, and responsiveness of the community (in my obviously-biased view, of course!).

A related concern, was (quoting the blog post):

> A small team of highly talented developers who can basically hold all of the code in their collective heads can make great progress while eschewing anything that isn’t just straight coding at the cost of making it more difficult for other people to contribute. Is that worth it? It’s hard to say. If you have to slow down Jeff, Keno, and the other super productive core contributors and all you get out of it is a couple of bums like me, that’s probably not worth it.

Julia as a whole has had 697 contributors right now. At the very core of the language, the parser and lowering code have well over 40 contributors, and code-generation (LLVM lowering) has over 60.

Re: Python’s Weak Performance Matters

#183
post #144
post #138

Earlier quoted context omitted.

Exactly, the ecosystem is the only reason IMHO why Python is "easy". I find I'm much quicker at developing C#/F# if the library is available on NuGet (which is often, but not always the case). They're both reasonably fast, too. Python only has a very large community and thus ecosystem to offer.

It can't be the only reason, or how would it ever have attracted such an ecosystem in the first place? Especially with its performance characteristics, and lack of corporate backing. No, Python was invented at a time when its closest competitor was Perl - and you need only compare typical Perl with typical Python to appreciate that Python really was a usability revelation. But that was nearly 30 years ago. I do think…

• the language is easy, very attractive for non-CS people

• the language is consistent (everything is an object, or a pointer to an object rather, you can only pass by pointer, scope and namespace that make sense all the time, etc...)

• you can learn it gradually, you can start using it even if you know only 10% of the language

• Amazing documentation. The official tutorial is easy to read, and by the time you are through with it you are fairly proficient with python

• error message that tells you exactly what the error is and where it is happening. A lot of other languages do that now, but that wasn't the case 25 years ago.

• inline help / magical docstring (`a=4 ; help(a)`)

• The REPL, especially being able to get into the REPL with your apps (`python -i`)

• batteries, again 25 years ago, no language came with such a huge standard library

• PEP 8, the grandfather to gofmt and rustfmt. Makes a big difference when working on a team

• the Zen (`import this`)

And of course as the language grew in popularity:

• the ecosystem (pypi.org)

• the amount of resources, from books to website, the answer to every questions you can think of in SO, very active and helpful mailing list/usenet group, active and helpful /r/Python

Re: Python’s Weak Performance Matters

#184

Earlier quoted context omitted.

Sure, here's an article from Microsoft describing how to access and view the source code for their library implementation, ASP/ADO/Forms/etc: https://weblogs.asp.net/scottgu/net-framework-library-source... That's from January 2008. This release is specifically what enabled Mono to really go to the next level, more than a decade ago. You can even see the little 'carve out' they made in the license to ensure Mono, in p…

The link you gave is indeed to view the source. It is not, however, an open source license, as characterized by the OSI or DFSG, nor free software as characterized by FSF. Here is the license - https://referencesource.microsoft.com/license.html . It contains obviously non-open source clauses like: > "Reference use" means use of the software within your company as a reference, in read only form, for the sole purposes…

From my perspective here, you're shifting the goal posts. In particular this conversation began with you apparently thinking that Mono lacked full implementations of ASP/ADO/etc. And I think that might justify the original comment that Linux implementations have been treated as second class citizens. I say might since ADO/ASP are not really part of .NET, but rather independent projects built on top of .NET. But in any case, that hasn't been an issue for many years now.

You have now shifted into claiming that your issue is that the licenses Microsoft released their code under were insufficiently permissive. I could argue against this, because it's at best misleading -- different code was released under different licenses. You can see an archive of one of the CLI releases [1] with the license it came with. Or this [2] is a blog from the Mono founder expressing thanks for Microsoft and ECMA members releasing code under open source licenses. That blog was from the huge 2.0 release, in 2008.

But ultimately that feels like a red herring, even if a fun one! You seemed to believe (as did the person I was responding to) that C# was a Windows only thing. That hasn't been the case for a very long time, and it is completely ridiculous now a days. In either case, it might seem we agree if you think that people are basing their views on obsolete information. But that then rather begs the question of why is that so especially the case here. If I was discussing e.g. javascript frameworks from the perspective of somebody 4 years out, let alone a decade, I'd be quite justifiably skewered. In other fields, 4 years ago things like TensorFlow did not even exist as public projects. 4 years is a very long time in software, so why are people so particularly slow on the uptake in this instance? Bias, bad marketing, something else?

[1] - https://web.archive.org/web/20060421052106/http://www.micros...

[2] - http://tirania.org/blog/archive/2008/Oct-06.html

Re: Python’s Weak Performance Matters

#185

Earlier quoted context omitted.

That is the _FAST_ version of the code (people keep saying "of course, it's slow", when it's the fast version). Here is an earlier version (intermediate speed): https://git.embl.de/costea/metaSNV/commit/ff44942f5f4e7c4d0e... It's not so easy to post the data to reproduce a real use-case as it's a few Terabytes :) * Here's a simple easy code that is incredibly slow in Python: interesting = set(line.strip() for line in…

I've also found that loops with dictionary (or set) lookups are a pain point in python performance. However, this example strikes me as a pretty-obvious pandas use-case: interesting = set(line.strip() for line in open('interesting.txt')) total=0 for c in chunks: # im lazy to actually write it df = pd.read_csv('data.txt', sep='\t', skiprows=c.start, nrows=c.length, names=['id','val']) total += df['val'][df['id'].isin(…

`isin` is worse in terms of performance as it does linear iteration of the array.

Reading in chunks is not bad (and you can just use `chunksize=...` as a parameter to `read_csv`), but pandas `read_csv` is not so efficient either. Furthemore, even replacing `isin` with something like `df['id'].map(interesting.__contains__)` still is pretty slow.

Btw, deleting `interesting` (when it goes out of scope) might take hours(!) and there is no way around that. That's a bona fides performance bug.

In my experience, disk IO (even when using network disks) is not the bottleneck for the above example.

Re: Python’s Weak Performance Matters

#186

Interesting article. I mostly agree. OP, could I ask a question? You mention 1TB files. Why do you guys at embl not use a database for this sort of stuff? I'd figure that with some proper indexing, I figure you could see pretty decent speedups just from that already.

For some things, we do. But databases are not magical and setting up a good table/index system &c is also work and there is overhead.

Thus, if we are talking about having (for example) a webservice where queries have a form that is known apriori, then it's a good solution. If you have output data from your processing that you will be slicing and dicing in different ways which you cannot predict ahead of time, then, they are not appropriate.

(Loading Terabytes of data into a database takes a while too).

Re: Python’s Weak Performance Matters

#187
post #85

Earlier quoted context omitted.

OP here. Speed is the main motivation, but total time is TimeToWriteCode + TimeToRunCode. Python has the lowest TimeToWriteCode, but very high TimeToRunCode. C++ has lowest TimeToRunCode, but high TimeTowWriteCode. Haskell is often a good compromise for me. Also, with Haskell, it can be very easy to take advantage of 20 CPU cores, while I don't have as much familiarity with high-level C++ threading libraries.

@ the OP - not to sound hostile, but you write code (like in the example here [1]) that is bound to be slow, just from a glance at it. vstacking, munging with pandas indices (and pandas in general), etc; in order for it to be fast, you want pure numpy, with as little allocations happening as possible. I help my coworkers “make things faster” with snippets like this all the time. If you provide me with a self-containe…

> Also, have you tried Numba?

Numba does not support dictionaries and has limited support for pandas dataframes (only underlying arrays, when convertible to NumPy buffers, if I understand correctly). This limits usefulness for many non-array situations, as well as some existing code-bases (the dictionary is fundamental in Python and typically used everywhere -- often for performance).

Re: Python’s Weak Performance Matters

#188

Earlier quoted context omitted.

> I see a 4x speedup on most of my slowest pure python workloads Heh, only 25..250 X to go. We did a direct line for line translation of some numerically intensive code from Python to C++ and saw a literal 1000X speedup. On other projects, it's been more like 100X slower. That says two things: first Python can be really slow, second, for some programs, Python doesn't really save on lines of code over modern C++. I've…

While I'm more of a pythonist than a C-ist, hearing "1000x speedup" and "line for line" to me implies that you aren't writing idiomatic python. Idiomatic python is (often) faster than not, and (often) more difficult to translate to lower level languages. As a simple example, list-comprehensions are faster than loops, and can't be line for line translated into C++.

> list-comprehensions are faster than loops

Are they really though? In my experience you'll gain a couple of percent because you're getting rid of that call to append, but that's hardly the orders of magnitude OP was looking for.

Re: Python’s Weak Performance Matters

#189
CPython is obviously the way it is on purpose, by design, and quite successful. Yet I still find it ironic that this "slow" interpreter is written in C, the go to, general purpose, "low level and a half" fast language, and that "C" is right there in the name. Not knowing anything else, I might expect a project with "C" at the front of the name would be at least fast-ish, and that the naming was intended to signal that.

Not a complaint, just an observation.

Re: Python’s Weak Performance Matters

#190
post #189

CPython is obviously the way it is on purpose, by design, and quite successful. Yet I still find it ironic that this "slow" interpreter is written in C, the go to, general purpose, "low level and a half" fast language, and that "C" is right there in the name. Not knowing anything else, I might expect a project with "C" at the front of the name would be at least fast-ish, and that the naming was intended to signal tha…

What? No. The c in cpython was not meant to signal performance. The c was added after alternative implementations were created to mean "the original/reference implementation". Python's competitors perl ruby and php are likewise interpreters implemented in c and are equally slow.

Anyway, what alternative did they have to implement a bytecode interpreter? C is portable and fast enough, especially with computed goto extension.

Post reply on HN