Live data from Hacker News

A beginners guide to using Python for performance computing

scipy.org

11–19 of 19 posts

Re: A beginners guide to using Python for performance computing

#11
post #8
post #6

Earlier quoted context omitted.

I often wonder why there is so little interest in a high level language that would compile to pure C++, with a syntax that makes code more succinct and friendlier, just like CoffeeScript compiles to JavaScript. We have a clear problem: higher level languages make programmers more productive, but they have performance issues. Because of those, developers that need performance fall back to C++ (look at videogame develo…

The weave bits, and Pyrex bits, found in this article are actually snippets of C/C++ written inline in the Python code (or very simplified Python), and able to access variables and such defined in the Python earlier (with caveats and when used with care), and later Python code is able to use the resulting compiled functions. So, it's high level Python code, except when you need to write low level, super fast code. Th…

It's just not possible to get it via the means you've described.

Why not? it might harder to implement than coffeescript, but ooc is proof that it's possible. Remove GC and dependent libs, and you still have a substancially higher level language than C (or C++).

As for ooc and garbage collection: How would you write a compiler for a high level garbage collected language without including a garbage collector?

I didn't say it should be garbage collected: Obviously it would have to come with some lower level semantics, like memory management. The coffeescript approach is it shouldn't have dependencies. By high level, I meant higher level than c++. it will always be lower level than a language like python, without dependencies we can't escape all the semantics underlying in the target language. That said like I mentioned before, memory management could possibly be handled by something similar to Apple's ARC (and to answer your last question, I don't think it's that black and white, look at how Apple is investing on ARC instead of GC; also GC's unpredictability make it a no-go for low level routines in videogames).

Re: A beginners guide to using Python for performance computing

#12
post #6
post #5

I once tried taking one of my C++ simulation codes (dealing with diagonalization of big memory-hogging matrices) and rewriting it using scipy. It was a lot slower. I don't have a good idea of exactly what situations you'll find scipy being significantly slower than C++. Maybe it was the fact I was playing with thousands-by-thousands arrays. But I do worry that there are folks out there burning millions of CPU hours r…

I often wonder why there is so little interest in a high level language that would compile to pure C++, with a syntax that makes code more succinct and friendlier, just like CoffeeScript compiles to JavaScript. We have a clear problem: higher level languages make programmers more productive, but they have performance issues. Because of those, developers that need performance fall back to C++ (look at videogame develo…

Shedskin compiles python to C++. It's getting quite good too. You can compile up to 10,000 lines within an acceptable time now. It is a subset of python though, so more dynamic things do not work.

You can also get it to compile as a CPython module. The latest release supports the mmap module, so it is possible to quickly share large array data without copying. Shedskin 1.0 is coming out maybe within a number of months after years of development.

Re: A beginners guide to using Python for performance computing

#14
post #12
post #6

Earlier quoted context omitted.

I often wonder why there is so little interest in a high level language that would compile to pure C++, with a syntax that makes code more succinct and friendlier, just like CoffeeScript compiles to JavaScript. We have a clear problem: higher level languages make programmers more productive, but they have performance issues. Because of those, developers that need performance fall back to C++ (look at videogame develo…

Shedskin compiles python to C++. It's getting quite good too. You can compile up to 10,000 lines within an acceptable time now. It is a subset of python though, so more dynamic things do not work. You can also get it to compile as a CPython module. The latest release supports the mmap module, so it is possible to quickly share large array data without copying. Shedskin 1.0 is coming out maybe within a number of month…

The C++ code it generates depends on a Python runtime, doesn't it? That means you can't use it to write - for example - C++ libraries.

Re: A beginners guide to using Python for performance computing

#15
Cython is pretty excellent for this sort of thing. The article contains a link to a Cython solution written by Travis Oliphant showing how concise (and fast) it is.

I just used Cython to speed up a non-vectorisable bit of code in a Gibbs sampling algorithm and I was impressed by the performance increase and ease of use.

Re: A beginners guide to using Python for performance computing

#16
post #11
post #8

Earlier quoted context omitted.

The weave bits, and Pyrex bits, found in this article are actually snippets of C/C++ written inline in the Python code (or very simplified Python), and able to access variables and such defined in the Python earlier (with caveats and when used with care), and later Python code is able to use the resulting compiled functions. So, it's high level Python code, except when you need to write low level, super fast code. Th…

It's just not possible to get it via the means you've described. Why not? it might harder to implement than coffeescript, but ooc is proof that it's possible. Remove GC and dependent libs, and you still have a substancially higher level language than C (or C++). As for ooc and garbage collection: How would you write a compiler for a high level garbage collected language without including a garbage collector? I didn't…

"Obviously it would have to come with some lower level semantics, like memory management."

Why? Garbage collection is nowhere near the top reason high level languages are slow, and there are GC languages that are fast.

"By high level, I meant higher level than c++. it will always be lower level than a language like python"

So, Java or C#, then? JIT languages that can be as fast as C/C++ in some circumstances, and faster in others (and slower in still others). And you don't even have to manage your own memory, so it's seemingly higher level than you're asking for.

I think GC is fine for high performance computing. Might not be fine for real time systems or games, but that's not what this article is about. It's about high performance scientific computing in Python. Having a GC sweep every now and then, even at unpredictable times, isn't going to cause things to come to a crashing halt or ruin the performance of a batch run.

PyPy might be something you'd enjoy reading about. It is an alternative implementation of Python (mostly written in Python, and using a JIT), with the goal of being fast like C. This may be the closest thing to a CoffeeScript like tool where the goal is to make a very high level language faster through translation or "compilation" or whatever.

Re: A beginners guide to using Python for performance computing

#17
I wrote an implementation of this using Python's array module, and doing manual index calculations for PyPy: http://www.reddit.com/r/programming/comments/hh8uj/a_beginne... . We'll soon have NumPy multi-dimensional arrays and thus can just run that code, but for now... we're doing OK I think.

Re: A beginners guide to using Python for performance computing

#18
post #11

Earlier quoted context omitted.

It's just not possible to get it via the means you've described. Why not? it might harder to implement than coffeescript, but ooc is proof that it's possible. Remove GC and dependent libs, and you still have a substancially higher level language than C (or C++). As for ooc and garbage collection: How would you write a compiler for a high level garbage collected language without including a garbage collector? I didn't…

"Obviously it would have to come with some lower level semantics, like memory management." Why? Garbage collection is nowhere near the top reason high level languages are slow, and there are GC languages that are fast. "By high level, I meant higher level than c++. it will always be lower level than a language like python" So, Java or C#, then? JIT languages that can be as fast as C/C++ in some circumstances, and fas…

"Obviously it would have to come with some lower level semantics, like memory management."

Why?

Once again: this approach means zero dependencies. And in a way, you answered it yourself:

Might not be fine for real time systems or games

And there are more examples where it might not be fine, here's a major one: building a C++ library. It's true, this article is not about real time systems (I do apologize for diverging...), but this language could be one solution to problems approached in this article (it would replace the C++ example).

I am interested in projects like shedskin or pypy, but they have dependencies. The bottom line to me is: a higher level language that compiles to C++ code with zero dependencies. Think about it: this language could be used for any application, deployed in virtually any codebase and any system.

Re: A beginners guide to using Python for performance computing

#19
post #8
post #6

Earlier quoted context omitted.

I often wonder why there is so little interest in a high level language that would compile to pure C++, with a syntax that makes code more succinct and friendlier, just like CoffeeScript compiles to JavaScript. We have a clear problem: higher level languages make programmers more productive, but they have performance issues. Because of those, developers that need performance fall back to C++ (look at videogame develo…

The weave bits, and Pyrex bits, found in this article are actually snippets of C/C++ written inline in the Python code (or very simplified Python), and able to access variables and such defined in the Python earlier (with caveats and when used with care), and later Python code is able to use the resulting compiled functions. So, it's high level Python code, except when you need to write low level, super fast code. Th…

"... people want the speed of low level and the convenience of high level languages."

http://basic-converter.org

Post reply on HN