A Python Compiler for Big Data

#1

Post by bluemoon » Tue, Dec 18, 2012, 12:41 AM UTC

A Python Compiler for Big Data
continuum.io

Re: A Python Compiler for Big Data

#2

Post by ezl » Tue, Dec 18, 2012, 1:37 AM UTC

I just want to point this out because I feel like there's a good chance a lot of people won't have gotten this far:

Because our implementation does not explicitly depend on Python we are able to overcome many of the shortcomings of the Python runtime such as running without the GIL and utilising real threads to dispatch custom Numba kernels running at near C speed without the performance limitations of Python.

Re: A Python Compiler for Big Data

#3

Post by freyrs3 » Tue, Dec 18, 2012, 1:42 AM UTC

ezl wrote: post #2

I just want to point this out because I feel like there's a good chance a lot of people won't have gotten this far: Because our implementation does not explicitly depend on Python we are able to overcome many of the shortcomings of the Python runtime such as running without the GIL and utilising real threads to dispatch custom Numba kernels running at near C speed without the performance limitations of Python.

Yes, using Numba we can just-in-time compile numeric Python logic straight down to machine code, so naturally we can achieve some pretty impressive numbers on kernel execution.

In case many people didn't reach the bottom here are the links to the repo and the docs. The project is still in early stages, but is public and released under a BSD license.

* http://blaze.pydata.org/docs/

* https://github.com/ContinuumIO/blaze

Re: A Python Compiler for Big Data

#4

Post by Caligula » Tue, Dec 18, 2012, 1:53 AM UTC

I read about continuum after the fellow who developed numpy left a few days ago to work on continuum. I am curious to see actual projects using continuum. So some sort of writeups.

Re: A Python Compiler for Big Data

#5

Post by omni » Tue, Dec 18, 2012, 2:48 AM UTC

Caligula wrote: post #4

I read about continuum after the fellow who developed numpy left a few days ago to work on continuum. I am curious to see actual projects using continuum. So some sort of writeups.

You're being downvoted because Travis Oliphant, the original author of Numpy, is also a co-founder of Continuum Analytics.

Re: A Python Compiler for Big Data

#6

Post by davidf18 » Tue, Dec 18, 2012, 2:50 AM UTC

It would be great to eventually have a GPU version as well (as in the cases of Matlab and R). I saw a brief demo of Matlab on a Mac Retina Pro 15 where the GPU version ran 30x the CPU version.

Re: A Python Compiler for Big Data

#7

Post by freyrs3 » Tue, Dec 18, 2012, 2:55 AM UTC

davidf18 wrote: post #6

It would be great to eventually have a GPU version as well (as in the cases of Matlab and R). I saw a brief demo of Matlab on a Mac Retina Pro 15 where the GPU version ran 30x the CPU version.

GPU support is definitely planned and already supported in NumbaPro[2]. Here's a video of Travis Oliphant's talk about targeting CUDA through Numba:

[1] http://www.ustream.tv/recorded/26973799

[2] https://store.continuum.io/cshop/numbapro

Re: A Python Compiler for Big Data

#8

Post by rpm4321 » Tue, Dec 18, 2012, 4:58 AM UTC

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython?

I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C.

From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

Re: A Python Compiler for Big Data

#9

Post by chadillac83 » Tue, Dec 18, 2012, 5:18 AM UTC

rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

LUA might be good for this too, it's pretty fast on it's own but from what I've read (not tested mind you) their C API is supposed to be pretty great.

http://benchmarksgame.alioth.debian.org/u32/which-programs-a...

http://en.wikipedia.org/wiki/Lua_(programming_language)#C_AP...

Re: A Python Compiler for Big Data

#10

Post by freyrs3 » Tue, Dec 18, 2012, 5:21 AM UTC

chadillac83 wrote: post #9

rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

LUA might be good for this too, it's pretty fast on it's own but from what I've read (not tested mind you) their C API is supposed to be pretty great. http://benchmarksgame.alioth.debian.org/u32/which-programs-a... http://en.wikipedia.org/wiki/Lua_(programming_language)#C_AP...

Lua has some areas where it excels in performance, but using Python you can leverage 10 years of work on numeric libraries that unrivaled in any other general purpose language. NumPy and SciPy are extremely powerful.